Abstract: PCI Hot Plug technology is a very exciting and new
technology focused on increasing server availability. Compaq
servers and drivers combine with Novell’s NetWare environments to
provide a solid PCI Hot Plug solution. This newest feature, PCI Hot
Plug Expansion, truly shifts the paradigm.
The expand feature of the PCI Hot Plug so greatly extends the
functionally provided by the earlier replace technology, that a very
careful examination is warranted. Superficially, the expand feature
not only allows new hardware to be added but also extends the
replace functionality to include replacement by dissimilar
technologies. When combined with duplexing, mirroring, and
pairing, the expand feature opens a completely new world of
opportunity to provide non-stop service, while upgrading, doing
maintenance, repairing, building, or migrating server components.
Help us improve our technical communication. Let us know what you think
about the technical information in this document. Your feedback is valuable
and will help us structure future communications. Please send your
comments to: Novell.Feedback@Compaq.com
PCI Hot Plug Expand Feature on Compaq Servers in Novell NetWare Environments2
Notice
The information in this publication is subject to change without notice and is provided “AS IS” WITHOUT
WARRANTY OF ANY KIND. THE ENTIRE RISK ARISING OUT OF THE USE OF THIS
INFORMATION REMAINS WITH RECIPIENT. IN NO EVENT SHALL COMPAQ BE LIABLE FOR
ANY DIRECT, CONSEQUENTIAL, INCIDENTAL, SPECIAL, PUNITIVE OR OTHER DAMAGES
WHATSOEVER (INCLUDING WITHOUT LIMITATION, DAMAGES FOR LOSS OF BUSINESS
PROFITS, BUSINESS INTERRUPTION OR LOSS OF BUSINESS INFORMATION), EVEN IF
COMPAQ HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
The limited warranties for Compaq products are exclusively set forth in the documentation accompanying
such products. Nothing herein should be construed as constituting a further or additional warranty.
This publication does not constitute an endorsement of the product or products that were tested. The
configuration or configurations tested or described may or may not be the only available solution. This test
is not a determination or product quality or correctness, nor does it ensure compliance with any federal
state or local requirements.
Product names mentioned herein may be trademarks and/or registered trademarks of their respective
companies.
Netelligent, Armada, Cruiser, Concerto, QuickChoice, ProSignia, Systempro/XL, Net1, LTE Elite,
Vocalyst, PageMate, SoftPaq, FirstPaq, SolutionPaq, EasyPoint, EZ Help, MaxLight, MultiLock,
QuickBlank, QuickLock, UltraView, Innovate logo, Wonder Tools logo in black/white and color, and
Compaq PC Card Solution logo are trademarks and/or service marks of Compaq Computer Corporation.
Microsoft, Windows, Windows NT, Windows NT Server and Workstation, Microsoft SQL Server for
Windows NT are trademarks and/or registered trademarks of Microsoft Corporation.
NetWare and Novell are registered trademarks and intraNetWare, NDS, and Novell Directory Services are
trademarks of Novell, Inc.
Integration Note prepared by Novell Integration
First Edition (November 1998)
Document Number ECG026/1097
ECG026/1097
PCI Hot Plug Expand Feature on Compaq Servers in Novell NetWare Environments3
Executive Summary
Compaq Peripheral Component Interconnect (PCI) Hot Plug servers are available, and they
implement the industry-standard PCI Hot Plug technology. Compaq helped develop this
technology as an extension to the PCI specification. Virtually all--old and new--PCI cards work
with the expansion capability and preserve your investment.
Upgrading online can now become routine.
For example when upgrading a disk system:
• Use the hot plug expand capability to connect the new controller
• Create volumes on the new drives
• Enable mirroring to replicate the data on the new disk system
• Break the mirror and default to the new disk system
• Remove the old disk system
• Add these old drives to the new system
Everyone and everything continues working during the process.
A detailed demo of this disk system upgrade/migration is available in the Compaq White Paper
"PCI Hot Plug Expand and Fibre Channel Migration in Novell NetWare Environments,"
ECG104/0998 at http://www.compaq.com/support/techpubs/whitepapers/.
Upgrading online to higher performance Network Interface Cards (NICs) for example, from 10
Mb/s to 100 Mb/s, is simple and non-disruptive. For a non-duplexed NIC, a single, free slot
allows a new NIC to establish service. For duplexed NIC pairs, two free slots are required to
establish the new service pair for the first network link. In both cases, the new NIC(s) are inserted
before the older cards are removed to ensure server availability on the network. Repeating the
process for each pair of NICs or each unpaired NIC on the server, provides an upgrade path
where no one is ever left without service. Everyone and everything continues working during the
process.
The most important features of these high-end servers are high reliability and fault tolerance. PCI
Hot Plug greatly increases the server’s ability to operate continuously, by providing the ability to
expand, replace, and remove components, while the server remains available to users. No powerdown, restarting, nor rebooting of the server is required. Virtually all of the clients actively
using the server continue to work uninterrupted, unaware of the upgrade or repair. Controllers and
devices can be added or replaced in this manner. On enterprise class servers, downtime is very
expensive, and the goal of PCI Hot Plug is to eliminate downtime. With the advent of PCI Hot
Plug with Hot Expand, everyone and everything continues working during the process.
ECG026/1097
PCI Hot Plug Expand Feature on Compaq Servers in Novell NetWare Environments4
Introduction to PCI Hot Plug
Compaq has been working for some time with key hardware and software partners to define and
develop PCI Hot Plug technology as an open, industry-wide standard. This open standard allows
addition, removal, and replacement of the PCI cards plugged into the Hot Plug slots of the PCI
bus(es) while the server continues to run and provide service to clients without interruption. The
goal of PCI Hot Plug technology is to eliminate downtime associated with adding, removing, and
replacing PCI adapters.
IMPORTANT:Expand is the new functionality being added to PCI Hot Plug. An earlier name
for expand is add. These terms are presently interchangeable.
The PCI Hot Plug specification is an extension to the original Peripheral Component Interconnect
(PCI) Local Bus Specification. One of the key features of the new standard is the compatibility
with existing PCI adapters. The fact that PCI Hot Plug is an open, industry-wide standard protects
your network investment:
• Better availability of products due to multiple sources for the product
• Better integration of products due to multiple vendors supplying interoperable products
• Less risk for your investment since current products that meet the original PCI
specification will continue to be produced, and future products will be developed to the same
standard
•Lower price for future products, due to both increased competition and larger markets
there is virtually no risk associated with this new technology.
Everyone wins with this type of open standard.
On any server, but especially on enterprise-class servers, the costs from the loss of productivity
due to a failed component can be enormous. When used in conjunction with fault-tolerant tools
(such as Redundant NICs, RAID controllers, Recovery Server options, and hot replaceable SCSI
drives), PCI Hot Plug is an important tool for high availability.. The fault tolerant tools keep the
server running when the failure occurs, and the PCI Hot Plug high-availability feature allows the
failed board to be replaced without interrupting access. When a redundant board in a Hot Plug
slot fails, there is no downtime. This is a result of the excellence that comes from matching
complementary technologies: high availability and fault tolerance.
IMPORTANT: When a fault tolerant component in a PCI Hot Plug slot fails, there is virtually
no downtime.
Following is a high-level example showing the advantages of the PCI Hot Plug adapter expansion
capability. When the primary NIC of a controller pair fails, switching to the secondary NIC
adapter occurs automatically, and the server continues to run. The administrator is notified by the
server of the NIC’s failure and then can
ECG026/1097
• Get a new NIC to replace the failed adapter.
• Open the PCI bus access coverplate.
• Turn off power to the selected slot by either of the following:
• Selecting the slot, from the Hot Plug main menu, which contains the NIC to be replaced,
and then choosing to remove the NIC.
PCI Hot Plug Expand Feature on Compaq Servers in Novell NetWare Environments5
•Or, on second-generation hot plug servers, pressing the service request button associated
with the failed card.
If your server is the second generation of the PCI Hot Plug servers, then there is a service
request button associated with each slot. These buttons are located just above the release lever
of each card slot, inside the access coverplate, on the back of the server. This button can be
used to indicate to the system that service for the slot is being requested. This button can be
used either in place of the console menus, or in conjunction with them.
Either of these steps will unload the device’s driver, and then power off the slot. The status of
the power down is updated on the main menu display and the slot’s LEDs.
Once the slot's green LED goes out to indicate no power, continue.
• Remove the failed NIC.
• Replace it with the new one.
• Power up the slot and card andreload the device’s driver.
• On a second-generation server, power up the slot by pushing the service request button.
• Or, on servers without the service request button, manually power up the slot from the
Hot Plug main menu.
The server will then be back at full health.
Duplexed disk adapter cards are replaced the same way.
PCI Hot Plug Components
A hot plug capable system requires
• PCI Hot Plug aware hardware
• Operating system supporting PCI Hot Plug: NetWare 4.11 and later, and NetWare 5.0.
• Open Data-Link Interface (ODI) version 3.3 for LAN drivers and protocols
An existing adapter board will be hot plug-able, if it is in compliance with the established PCI
specification and has hot plug device driver support. To ensure hardware support and
compatibility, each of the following hardware components must be provided for each PCI Hot
Plug expansion slot:
PCI Expansion Boards – Most existing PCI expansion boards are fully functional in any system
that has hot plug capabilities. However, for an existing PCI board to be hot plug capable, the
board must meet all PCI Hot Plug specification guidelines. In addition, in order for hot plug to be
functional, the device also needs software support, including hot plug-aware (NWPA or ODI
v3.3) device drivers.
ECG026/1097
PCI Hot Plug Expand Feature on Compaq Servers in Novell NetWare Environments6
Note: Some apparently fully functional PCI boards, which work in non-Hot Plug machines, may
fail in PCI Hot Plug machines due to their not complying completely with the PCI specification.
Specifically, some manufacturers did not implement the power-level sensing on two of the PCI
pins, but defaulted to using the bus' pull-up resistors. This will cause a failure on PCI Hot Plug
systems on which these pins are used to determine whether a card is present or not. By installing
the card and not relying on the auto sensing of the software , a workaround is possible. In this
way, an otherwise unusable card can be finessed into working.
IMPORTANT: Use Compaq PCI Hot Plug drivers to avoid problems.
Slot Release Lever – The slot release lever holds the boards in the I/O expansion slots. When the
lever is closed, the hot plug driver can apply power to the slot through the operating system
software. If the lever is opened, power is removed from the slot.
CAUTION: You should always turn off power to the slot before opening the lever.
Expansion Slot LEDs – Each PCI Hot Plug slot has two LED indicators above the slot release
lever. When the green LED is on, it indicates power has been applied to the slot. When it is off,
there is no power applied to the slot. The amber LED indicates that a condition has been detected
that requires corrective action for normal operation to continue, such as a problem with the slot,
the board, or the device driver.
PCI Slot Service Request Button− This feature is available only on the newer generation of
PCI Hot Plug Servers. Pressing the button signals the software that attention is needed. Use of
this button is optional. The same functionality is provided through the software interface, but this
request button is easier to use. Each slot has its own button indicating which slot is to be
addressed by the supporting software. The power state of the slot and of the presence or not of a
board determine the action performed, when the service request button is pressed. Table 1 below
shows the actions that occur when the service request button is pressed.
Table 1 Logic Table for Service Request Button
Power- OffPower-On
Board PresentSoftware powers on the slot,
checks and identifies the
board, and automatically loads
or enables the device drivers
Board Not PresentNo ActionNo Action
PCI Hot Plug/NetWare Installation
PCI Hot Plug should be installed using the latest version of the SmartStart CDs available from
Compaq. PCI Hot Plug is not by default an option loaded during installation and must be selected
in the options for the SmartStart setup.
Software unloads or disables
drivers and powers off the slot
ECG026/1097
PCI Hot Plug Expand Feature on Compaq Servers in Novell NetWare Environments7
PCI Hot Plug/NetWare Integration
Novell Configuration
Manager Console
Novell Event Bus
Novell Configuration
Manager
PCI Expansion Board
System Hot Plug Hardware
Figure 1. Novell NetWare Hot Plug Architecture
ODINWPA
LAN DriverStorage Driver
Installation
Tools
Other NetWare
Loadable Modules
OEM Specific
System Bus
Driver
The PCI Hot Plug architecture, shown above in Figure 1, leverages the inherent modularity of
NetWare to minimize changes required of third-party adapter board software. The system relies
on a new central component, the Novell Event Bus, which facilitates communications between
different software modules. The Event Bus is implemented as a NetWare Loadable Module
(NLM), allowing implementation of PCI Hot Plug on existing versions of NetWare.
Listed below are the components of the PCI Hot Plug architecture for NetWare:
• Novell Event Bus (NEB)
• Novell Configuration Manager (NCM)
• Compaq OEM-Specific System Bus Driver (CPQSBD)
• Novell Configuration Manager Console (NCMCON)
• Device drivers
• ODI-compliant network boards
• NWPA-compliant storage device boards
• Installation tools
ECG026/1097
PCI Hot Plug Expand Feature on Compaq Servers in Novell NetWare Environments8
Novell Event Bus
The central component of the PCI Hot Plug software architecture is the Novell Event Bus Module
(NEB.NLM). This module facilitates communication between the administrator, the software
modules, management agents, and device drivers, providing the foundation for the software
architecture. The Event Bus is implemented for the time being as an NLM, allowing
implementation of PCI Hot Plug on existing versions of NetWare 4.11 and later, and 5.0. Future
versions will integrate the Event Bus into the operating system.
Novell Configuration Manager
For PCI Hot Plug servers, the Novell Configuration Manager (NCM.NLM) manages the hot plug
system and ensures proper system behavior. Its primary function is to sequence hot plug
operations to ensure stability of the system. To achieve this function, the Configuration Manager
properly shuts down a board's device driver before removing power from the slot. The
Configuration Manager also maps hardware resources to device driver software. Future releases
of NetWare will have an enhanced Configuration Manager to encompass other technologies in
addition to PCI Hot Plug.
System Bus Driver
Compaq is providing a system bus driver specific to the PCI Hot Plug hardware implementation.
The System Bus Driver (CPQSBD.NLM) abstracts the platform-specific implementation of PCI
Hot Plug in the ProLiant family of servers. This driver controls the system hardware necessary to
perform hot plug functions and is responsible for the following:
• Enabling or removing power from a hot plug slot upon request.
• Configuring hardware and assigning resources like memory, I/O ports, and interrupts for PCI
boards in hot plug slots.
Novell Configuration Manager Console
The Novell Configuration Manager Console (NCMCON.NLM) is necessary to perform hot plug
functions in NetWare systems. This NLM provides the interface for controlling all hot plug
operations. In addition, it displays slot configuration details, including board driver information
and status.
The Configuration Manager Console consists of two screens: the Main Menu and Slot Detail
Information screen. The Main Menu displays a list of all PCI slots in the system.
For each slot, the Main Menu displays the following information:
• Slot number
• Slot hot plug status
• Product name or description of the board
ECG026/1097
•Status of the slot/board
The Slot Detail screen displays information about a specific slot and the board in that slot. This
information includes the following:
•Slot number
PCI Hot Plug Expand Feature on Compaq Servers in Novell NetWare Environments9
• Board name or description
• Bus number associated with that slot
• Slot/board hot plug status
• Device driver information associated with the board
Pressing F2 activates the Slot Options menu, which allows the administrator to take actions such
as removing or replacing the board in that particular slot.
32-Bit ODI LAN Device Driver Requirements
The NovellOpen Data-Link Interface(ODI) specification documents the driver architecture for
network devices. The ODI specification has recently been modified to encompass PCI Hot Plug
technology. Because of the inherent modularity of NetWare, the changes required in third-party
board device drivers are minimal.
The ODI architecture standardizes the support for network PCI Hot Plug drivers by providing a
translation layer between the specific device drivers and the operating system. This translation
layer, called ODINEB, abstracts the ODI device drivers (hardware-specific modules) from the
PCI Hot Plug system. Device drivers compliant with the modified ODI specification are PCI Hot
Plug compatible, but will still work in non-hot plug environments.
To obtain PCI Hot Plug functionality, LAN drivers need to be compliant with either the
Assembly HSM 3.31 specification or the CHSM 1.11 specification.
NWPA Device Driver Requirements
NetWare Peripheral Architecture (NWPA) is the new API for developing storage device drivers
for Novell’s operating systems. It now replaces Device Drive Functional Specification (DDFS).
PCI Hot Plug device drivers work only under NWPA. The PCI Hot Plug software requires as a
minimum the NetWare 4.11 operating system. It is not recommended to mix DDFS and NWPA
device drivers on the same system.
NICs, Redundant NIC Pairs, and Upgrade
Concepts
Individual NICs can be upgraded in a very straightforward fashion with the PCI Hot Plug Expand
feature. Simply place the new higher performance NIC in a slot, connect the network, and power
up/load the device driver. With both boards powered up and ready to run, the only remaining
issue is the software protocol assignments. For versions of NetWare before 5 and Client 32, this
requires but a moment of service downtime, while the old protocol bindings are removed from the
old NIC and then reassigned to the new NIC. Only the clients using the old NIC are affected
during this software switchover; all other clients will not be interrupted. With NetWare 5 and
Client 32, the command to load the new bindings will automatically unload the old bindings, so
no clients will be affected. The old card can then be removed.
ECG026/1097
In order to upgrade NIC Pairs using the expand feature of PCI Hot, the concept of Redundant
NIC Pairs and its implications must first be understood. Network Interface Cards (NICs) can be
paired as a primary NIC and a secondary NIC in a special relationship called a Redundant NIC
Pair. In this relationship under NetWare, both NICs are connected and operational, but only one
NIC in the pair operates at a given time. The first NIC of a pair to be loaded is defined as the
secondary NIC. When the second NIC, the primary NIC, is loaded, it is linked to the secondary
PCI Hot Plug Expand Feature on Compaq Servers in Novell NetWare Environments10
NIC and completes the establishment of the Redundant NIC Pair. Both NICs in a Redundant NIC
Pair must be identical and be controlled by the same driver. When the primary NIC of the pair is
detected to have failed, the device driver will have the secondary NIC take over control and
continue to provide service. This Redundant Pairing is at the heart of the fault tolerant NIC
service. The primary NIC is the device initially used to provide service, and the secondary is held
in reserve. It provides service only in case of a failure. Currently, Compaq NETFLEX−3 adapters,
Netelligent adapters, and Netelligent TLAN-based adapters can form Redundant NIC Pairs; the
NICs used in a pair must be identical. The process of pairing the Redundant NIC Pairs is done
during the loading of the device drivers with keywords PAIR and SECONDARY, as follows:
The management of pairs is done through the CPQONLIN.NLM utility.
For a more detailed look at Redundant NICs and how they work, please see the NSSD
Documentation, the section entitled "Advanced Network Fault Detection and Correction".
The goal of forming Redundant NIC Pairs is to continue to provide service in the event of any
failure. Most failures are not NIC failures, but turn out to be disconnected wires. If a failure such
as a broken connector or an actual NIC failure occurs to a Redundant Paired NIC, the secondary
NIC will take over and provide a continuation of the service. The device driver for Redundant
NIC Pairing will notify the system administrator of the failure, both by a console message and by
pager (if this has been set up). The failed NIC can then be replaced (via the PCI Hot Plug replace)
or the connector can then be repaired. A NIC in a Redundant NIC Pair can only be replaced with
an identical NIC.
A Redundant NIC pair must be treated as a single entity, and not as two individual NICs. When
an upgrade is made, say from 10-Base-T to 100-Base-T service, the old pair of NICs and their
two slots cannot be used until the device driver for both NICs is unloaded. This dissolves the
Redundant NIC Pair as well as the service provided by the pair. Therefore, when an upgrade is
desired, the new service must be established first.
This requires two free Hot Plug slots for the new NICs being added to the server with the hot plug
expand feature. Once the new NIC pair is installed and the connection to the same network
established at the higher frequency, the old Redundant NIC Pair is ready to be removed. The
protocol mapping must be switched first, and then the old NICs drivers can be unloaded and the
NICs removed from the system. This will free two slots, and then another pair of NICs can be
upgraded. The new hot plug expand functionally will allow all NICs to be upgraded, one pair at a
time.
SCSI Upgrade Concepts
SCSI systems can be upgraded online with the new expansion capability. These upgrades involve
(1) coordinating new hot plug expand functionality with the disk drive; (2) duplexing and
mirroring; and (3) moving, removing, and adding SCSI devices. Several types of upgrades are
presented:
• Addition of new non-hot plug-able devices and drives to a SCSI bus
• Addition of new hot plug-able drives and controllers
ECG026/1097
•Replacement of entire disk systems
PCI Hot Plug Expand Feature on Compaq Servers in Novell NetWare Environments11
Non-Hot Plug-able SCSI
Non-Hot Plug-able SCSI devices present a problem for upgrading. It is not advisable to plug one
of these devices, such as a disk, tape, or CD-ROM, into a SCSI bus that is active. Doing so can
cause problems ranging from a mere disruption of service, to actually damaging or destroying the
SCSI controller or other connected devices. Therefore, the SCSI bus must be inactive to add nonhot plug-able devices.
The expand capability of PCI Hot Plug means that new activities can be done dynamically on the
PCI Hot Plug bus slot. Drivers can be unloaded and slots can be powered down, deactivating the
SCSI bus. Afterwards, devices can safely be moved, removed, or added. Power can then be
restored and the drivers for the new arraignment of devices brought into service. Care must be
taken to insure that the devices on the original SCSI bus are not providing a critically needed
service. If they are, you will be unable to power down the SCSI bus to remedy the situation
without causing a major disruption of the system.
In the case of adding disks to a system where an existing SCSI controller will be used with the
new disk drives, the devices already on the SCSI bus must first be examined. It is possible to
only marginally disrupt clients by removing seldom-used or special devices from service during
the upgrade, using the method outlined above. If there are disks on the SCSI bus and they are not
mirrored, removing them from service for only a short time during the upgrade is preferable to
downing the entire server for the upgrade. Some diminished level of service can be provided
during an upgrade of this type.
An upgrade to a mirrored SCSI system can be done without disrupting service, but some special
care and time must be taken. Adding new drives should be added serially, beginning with the bus
where the primary data set resides. Removing the SCSI device driver and powering down the PCI
Hot Plug slot that contains the primary data set's SCSI controller will cause the service provided
by the primary data set to fail over to the secondary data set. Once the new disks are in place and
powered up, the data sets can be re-synchronized and service can be returned to the primary data
set.
A second set of drives can be added to the bus containing the mirrored data set in similar fashion,
and the data sets once again re-synchronized. The new disks can then be put into service by
expanding both data sets. By doing this carefully, service to clients is not disrupted. Evaluation of
this method must include a risk analysis of having a single point of failure during the two times
when the data sets are not synchronized.
Addition of Controllers and Drives
This is a straightforward use of the PCI Hot Plug Expand feature:
• Add the new controller card to the empty and powered off slot
• Plug in the new drives to the SCSI-bus expansion box
• Connect the SCSI cable from the bus to the new controller card
• Power up the expansion box and drives
ECG026/1097
•Power on the PCI Hot Plug slots containing the SCSI controller(s)
The PCI Hot Plug software should recognize the new controller(s) and automatically load the
SCSI drivers.
PCI Hot Plug Expand Feature on Compaq Servers in Novell NetWare Environments12
Disk System Replacement
Summary
The replacement of a disk system can be accomplished without loss of service. Further, the
existing old disks can be recycled into the disk array, in this example a Fibre Channel (FC) array.
The only caveat to this scenario is that the boot partition of a Novell system cannot be moved.
This online replacement is done in the following easy steps:
1. Attach New Fibre Channel Hardware
•Arrange and connect the Fibre Channel hardware: hub (optional), FC Expansion box,
drives, and cables
• Place a Fibre Channel controller card into a powered-down PCI Hot Plug slot.
• Attach the Fibre Channel cable between the card and Fibre Channel hardware.
• Power the Fibre Channel hardware up.
2. Hot Plug Expand the Fibre Channel
•Once the Fibre Channel hardware and disks have come up to speed, perform a hot plug
expand on the Fibre Channel PCI controller card that is in the powered down PCI slot.
Assuming the drivers are already in SYS:SYSTEM, the Novell Configuration Manager
Console utility (NCMCOM.NLM) will first detect the new PCI card and then will auto
search for the corresponding correct drivers and load them.
3. Configure the Disk Array and Mirror the Data
•Configure the disks in the Fibre Channel expansion box into array(s) using the Compaq
Online Configuration Utility (CPQONLIN.NLM.)
•Using the Novell Install utility (INSTALL.NLM), create a NetWare partition on the
array(s)
•Using the Novell Install utility (INSTALL.NLM), mirror the existing disk array(s) onto
the new Fiber Channel array(s), and allow them to get synchronized (this may take a
while, depending on the amount of data involved in the process)
Your data migration is now complete. A crucial element in any disk replacement operation is to
guarantee that no data is lost. You should of course make a backup (recommended by Compaq),
or you can rely on the performance of the array system. A RAID 5 system is very reliable. Up to
this point in the operation that we are describing, there has not been, at any point, any additional
chance of data loss. The mirroring has only exactly replicated the data. Once you are comfortable
that both the new and old disk drives and disk systems are running well (or you have backed up
the data), then the next steps can be done safely.
4. Migrate old disks
•Use the Novell Install utility (INSTALL.NLM) to delete the original device(s) from the
mirror
•Move the original hard drives into the Fiber Channel system
ECG026/1097
PCI Hot Plug Expand Feature on Compaq Servers in Novell NetWare Environments13
•Use the Compaq Online Configuration utility (CPQONLIN.NLM) to add these drives
(that you just removed in the previous step) either to create a new array or to expand the
existing Fibre Channel storage array.
Now the old drive system has been replaced with a new larger capacity system, without loss of
service, data, and without having to discard the old disks.
Step by Step Detail of the Procedure
The previous section presented an outline of the procedure for the replacement of a disk system.
For a complete detailed description of this process, please see the Compaq White Paper
ECG1040998, "PCI Hot Plug Expand and Fibre Channel Migration in Novell NetWare
Environments", Oct. 1998.
Performing Hot Plug Actions with NetWare
Compaq PCI Hot Plug and the Novell Configuration Manager work together by providing the
following hot plug actions:
•View slot detail information – This allows you to see detailed information about each PCI
slot, see Figure 2, giving details not only on what the system thinks is the current state of the
slot, but also any board present in the slot.
•
Figure 2. Novell Configuration Manager – System Information
ECG026/1097
PCI Hot Plug Expand Feature on Compaq Servers in Novell NetWare Environments14
•Hot removal of boards – You can manually shut down and permanently remove a board: see
Figure 3.
Figure 3. Novell Configuration Manager – Switch open for PCI Slot 8
•Hot replacement of boards – This allows you to replace a failed board with an identical
board. This action applies power to and configures the slot, see Figure 4.
PCI Hot Plug Expand Feature on Compaq Servers in Novell NetWare Environments15
•Hot expansion of boards – You can insert a new board into an empty slot, load the
associated device driver, and power up the slot to provide a new service. See Figure 5.
Figure 5. Novell Configuration Manager – New adapter available, powered on with drivers loaded.
Discussion on Drivers
Compaq strongly recommends use of their latest NWPA and ODI drivers.
Note: For additional information on NWPA, see: “Migrating to Novell’s NetWare Peripheral
Architecture (NWPA) Drivers,” a Compaq White Paper, August 1997.
The following discussion is about drivers in general; what is available, and the problems and/or
advantages associated with each of them. The manner in which PCI Hot Plug is handled will vary
from device to device and from class to class. Drivers fall into the following classes:
•Persistent Drivers. These drivers remain loaded and present throughout the replacement,
requiring no attention from the administrator to load and unload the instance of the driver.
These drivers do, however, require the administrator to switch power to the slot and enable
the slot.
•NWPA and ODI Drivers that support instance unloading. During the replace, drivers in
this class, like the NIC driver, must be unloaded. After the card is replaced, the driver must be
reloaded. With instance unloading, a feature supported by NWPA, all other instances of the
driver remain intact—except for the driver of the single specific slot being replaced. The
administrator is prompted to remove the driver, and later after the replacement of the card, is
prompted again to reinstall the driver’s instance
ECG026/1097
•NWPA and ODI Drivers that do not support instance unloading. If the device driver is
such that it does not support the individual instance unloading of the driver, then all instances
of the device must be unloaded. Although NWPA supports instance unloading, the individual
PCI Hot Plug Expand Feature on Compaq Servers in Novell NetWare Environments16
drivers are called to perform the unloading, and the way they were written determines if they
support this feature. Instance unloading support is optional for NWPA drivers.
•DDFS Drivers. The entire driver with all of its instances must be unloaded.
In the first two classes of drivers and in the replacement of a failed card, none of the users on the
server will notice the failure or the replacement of the fault-tolerant devices. Their server takes
advantage of the high-availability capability provided by PCI Hot Plug. Both they and their data
will continue to work, uninterrupted. All Compaq NWPA drivers fall into these categories.
The DDFS class drivers and poorly written NWPA drivers that don’t support instance unloading
of the device present problems. These drivers will disrupt service for everyone on the server using
the type of device being replaced since all instances of the device driver must be unloaded. The
solution to this problem is to replace these drivers with the new NWPA Compaq-supplied drivers
that support instance unloading. Any fault-tolerant device, supported by drivers that don’t support
instance unloading, cannot be used to take advantage of the PCI Hot Plug’s high availability and
should be replaced. ALL Compaq NWPA drivers fully support instance unloading.
Note: DDFS drivers are currently supported by NetWare 3.x and NetWare 4.x. NetWare 5 does
not support DDFS.
Hardware Requirements
Compaq PCI Hot Plug servers (ProLiant 6500 and ProLiant 7000) offer independent control of
the power and logic for each slot on the PCI Hot Plug bus(es). Each slot also senses the presence
of the adapter card and the state of the card latch. Every slot is also equipped with LED displays
to indicate the power and logic condition of the slot to the user. In order to implement PCI Hot
Plug, the software must be aware of these slot features and must provide for the interaction
between the hardware and user. There are no additional hardware requirements on the PCI
adapter cards. Currently available and future PCI adapter controller cards will function in
Compaq server PCI Hot Plug slots, as per specification.
Software Requirements
PCI Hot Plug Support requirements:
• Novell NetWare Support Pack 5 (or later) for NetWare 4.11 and later, or NetWare 5.0
• Novell Support Software Diskettes (NSSD) version 3.20 or later from Compaq
• ODI version 3.31, or later, compliant network support
• NWPA version 2.32 or later compliant storage support
The PCI Hot Plug software supports the NetWare 4.11 and later operating systems as well as
NetWare-SMP. PCI Hot Plug is not supported on NetWare SFT-3, NetWare for OS/2, or previous
versions of NetWare.
ECG026/1097
PCI Hot Plug Expand Feature on Compaq Servers in Novell NetWare Environments17
The latest NSSD can be created via Compaq SmartStart CD or
obtained from the Compaq website (http://www.compaq.com).
The latest versions of Compaq drivers are always available at
the Compaq website (http://www.compaq.com).
The latest Novell software and patches are on Novell NetWare,
via CompuServe, or the World Wide Web
http://www.novell.com or http://support.novell.com
The latest Compaq software and patches can be obtained from
the Compaq website: http://www.compaq.com
Contingencies
(Known at the time of publication)
Compaq provides everything necessary for the
installation of the PCI Hot Plug software
through SmartStart release 4.0 or later.
SmartStart provides an assisted and highly
automated installation that provides excellent
documentation on PCI Hot Plug functionally.
The versions of software on the Compaq
SmartStart CD are the latest available at the
time of this writing. As newer versions of the
software become available, they should be
used.
Removing Adapters
On ProLiant 6500 and ProLiant 7000, when removing an adapter that has the NWPA storage
driver loaded, the driver normally unloads automatically for that adapter. However, if
NWPA.NLM was loaded before NEB.NLM, the driver does not automatically unload and the
Novell Configuration Manager Console (NCMCON) displays the following message:
“Driver for adapter in Slot # is currently loaded.
Unload driver and select REMOVE again.”
Solution:
Edit the STARTUP.NCF file to ensure that NWPA.NLM is loaded after NEB.NLM. Place any
lines that autoload NWPA.NLM after the NEB.NLM. load statement. Loading any Host Adapter
Modules (*.HAM) will cause NWPA.NLM to autoload.
Server Fails to Boot After Installation
If for some reason the server fails to boot after installation, then you will need to manually
rename the SERVER.OLD file to SERVER.EXE to bring the server back up.
When the software is installed, SERVER.EXE will be updated with the current version of
LOADER. Some customers have indicated that the modified version of SERVER.EXE becomes
corrupted and displays the following message: "LOADER-4.10-19: Error reading load fileC:\NWSERVER\SERVER.EXE ". This corruption has been linked to the DIE_HARD virus.
ECG026/1097
Solution:
PCI Hot Plug Expand Feature on Compaq Servers in Novell NetWare Environments18
•Scan for viruses on the DOS boot drive of the file server, as well as any locations to which
the patch kit may have been copied. Novell has ensured that the virus has not been
encapsulated in the compressed Novell Support Pack file.
•Rename SERVER.OLD (or MSERVER.OLD if running NetWare SFT III) to SERVER.EXE (or
MSERVER.EXE).
• RUN LSWAP.EXE. USAGE: LSWAP [LoaderFilePath [ServerFilePath]]
• Run the newly modified SERVER.EXE (or MSERVER.EXE).
Files Cannot be Copied to the DOS Partition
If you remove DOS, you must restart the server with DOS active in order to install the NetWare
Support Pack. This is because files cannot be copied to the DOS partition unless DOS is active.
Installing Updates to English-Only Services
•In order to install updates to services that are English-only, the server language must be set to
English. The English-only products in the Support Pack are NetWare MPR 3.1 and the files
under the ENGLISH directory.
Tips
NetWare Connect v2.0
•If NetWare Connect v2.0 is installed on the server, install the NetWare Support Pack first.
After installing the Support Pack and before restarting the server, install the NetWare
Connect 2.0 patch 2.0.30 found in the following three files: NWC206.EXE, NWC207.EXE,NWC208.EXE. Refer to TID 2915666 for more specific instructions.
The UXP200.EXE and NIP200.EXE files are included in this NetWare Support Pack.
•It is possible to create other directory structures to accommodate your needs. When extracting
the NetWare Support Pack file, the \COMMON\PK411 and \COMMON\SYS directories are
created. These directories are available for installing other files, such as NLM programs. The
script file will copy files from the \COMMON\PK411 directory to the location where
SERVER.EXE or MSERVER.EXE resides on the server PK411 drive. The script file will also
copy files from the \COMMON\SYS directory to volume SYS:. It is possible to create other
directory structures to accommodate your needs.
• Example:
• This example illustrates how this concept works. By default there are two directory
structures: \COMMON\PK411 and \COMMON\SYS\SYSTEM. If a disk driver needs to be
updated on the boot directory (C:\NWSERVER), copy the .DSK file to the
\COMMON\PK411 directory. If an NLM program needs to be updated on SYS:\SYSTEM,
copy the .NLM or .LAN file to the \COMMON\SYS\SYSTEM directory.
ECG026/1097
•If a file needs to be updated in any other path, such as SYS:\ETC, create a directory under
MON\SYS and copy the file to directory \COMMON\SYS\ETC.
PCI Hot Plug Expand Feature on Compaq Servers in Novell NetWare Environments19
CAUTION: Do not delete any files shipped with the NetWare Support Pack with the intent that
they will not be installed. Various NLM programs may have dependencies on each other, and by
deleting one NLM program you may cause another one to fail, resulting in a server ABEND. On
the other hand, you may add files so that the NetWare Support Pack installation program will
install them.
•
• Additional Information about the Support Pack installation:
• The Support Pack files are copied to the server.
• LSWAP will automatically update SERVER.EXE with a new LOADER.EXE.
• A record will be added to the Btrieve PRODUCTS.DAT file, in the "Installed Products"
section, and the following will appear: "SPACK v3.0 Support Pack for NetWare 4.11 (6/97)".
Any other product updates will also be listed.
•The Support Pack installation detects the current server configuration and installs the
appropriate files. If the server configuration is changed, you will need to reinstall the Support
Pack for the new configuration, for example, changing from NetWare SFT III to NetWare
•Novell tested the v3.0 NetWare Support Pack with the following Novell products or services:
Web Server v2.51; NetWare for SAA v2.2; Unix Print Services v2.11b; ManageWise v2.2;
NetWare IP v2.2; GroupWise v5.1; NetWare Connect v2.0 & v3.0; SBACKUP.NLM (test for
any SMS compliant backup); NetWare for Macintosh v4.11; NIAS v4.0 (includes MPR 3.1);
and MultiProtocol Router v3.0. Novell has encouraged third parties to test their products with
NetWare Support Pack by providing them with an Alpha copy and advance notice of the
release schedule
•If the backup option is selected from the menu, files that will be overwritten by installing the
Support Pack are backed up to SYS:SYSTEM/!BACKUP.SP3.
CAUTION: Do not do the following if any products or patches have been installed after the
installation of the support pack.
• The server can be restored to its original state by doing the following:
• Roll Back SYS Volume:
• From a workstation, map a drive to the SYS volume, go to that drive, and change the
SYSTEM\!BACKUP.SP3\SYS directory.
•Copy all of the files in this directory and its subdirectories the root of the disk SYS (i.e.
xcopy *.* \ /s /v).
• Roll Back BOOT FILES:
• From a workstation, map a drive to the SYS volume, go to that drive, and change the
SYSTEM\!BACKUP.SP3\BOOT directory.
• RCONSOLE to the server from the same workstation.
• From Rconsole, press ALT <F1>, to bring up the Rconsole menu.
• Choose the option "Transfer Files to the Server".
• Enter the drive mapped to SYS:\SYSTEM\!BACKUP.SP3\BOOT and press return.
ECG026/1097
PCI Hot Plug Expand Feature on Compaq Servers in Novell NetWare Environments20
• Enter the path to the boot directory on the DOS partition (i.e. c:\nwserver).
Handling a Failed Replacement
In a perfect world, everything works fine the first time. During a replacement operation, it is
possible that a NIC that should work, will not. There can be several reasons for this type of
failure. In the ROM identification of the board, there are several fields that identify the
manufacturer of the chip, the manufacturer of the board, and compatibility of the board with other
boards and drivers. If the information contained here does not match exactly what the driver
expects, the replacement operation will fail. If the new board itself is bad, the replacement will
fail. In these and similar situations of failure, this section describes the messages showing the
failure and how to recover.
PCI Hot Plug Troubleshooting
Server locks up or reboots when using NCMCON.NLM to replace a PCI adapter:
The NBI.NLM (v1.62) support module requires DOS support to be present while the server is
running. This support module is used by the PCI Hot Plug software and will cause the server to
hang during Hot Plug Replace or Expansion operations if DOS support is not present. The
"REMOVE DOS" command should be removed from the AUTOEXEC.NCF if it is present, and
should not be issued at the server console. The "SECURE CONSOLE" command, which
performs the same functions as "REMOVE DOS," may be used instead, as it does not affect
NBI.NLM.
Novell Configuration Manager Error Messages
Unable to allocate memory
The module was unable to allocate memory for internal use. Unload other modules or add more
memory to the system to prevent this error from occurring.
Error calling NBI.NLM. Version of NBI.NLM may be old.
Upgrade to a newer version of NBI.NLM.
Error registering with the Novell Event Bus.
Either this version of the Novell Event Bus is not compatible with this version of the Novell
Configuration Manager, or there is a conflict with a System Bus Driver.
Removing an Adapter Error Messages
The following error messages may appear when you remove an adapter:
Volume(s) mounted for adapter in Slot #. Do you wish to force removal?
This prompt displays when a disk controller is attached to a drive that is being used. Select “Yes”
to (a) force the drive to be dismounted and (b) force the driver for the adapter to be unloaded.
ECG026/1097
PCI Hot Plug Expand Feature on Compaq Servers in Novell NetWare Environments21
Driver for adapter in Slot # is currently loaded. Unload driver and select
REMOVE again.
This error occurs if a REMOVE was selected for an adapter where the driver is loaded and does
not support automatic unload. You must go to the main console and unload the instance of the
driver for the adapter to be removed. The name for the driver can be obtained from the Slot Detail
screen. Once the driver has been unloaded, the adapter shows a status of READY on the main
menu, and the driver shows a status of INACTIVE on the Slot Detail screen. The REMOVE
option can then be selected to remove the adapter.
REMOVE action not supported for the adapter in Slot #.
This message occurs when either the Configuration Manager or the System Bus Driver cannot
shut down the slot. This may be caused by the special cases of multiple card sets and the inability
of the Configuration Manager or System Bus Driver to perform the remove. If hot plug removal
is not supported for these adapters, they must be removed by powering down the entire system.
Replace not supported for adapter in Slot #. Do you still wish to force
removal of adapter?
The PCI Hot Plug software cannot configure some adapters, for example, a PCI to an ISA bridge;
therefore those adapters can only be removed and not replaced.. Selecting “Yes” at the prompt
allows the adapter to be removed but not replaced.
Remove for Slot # was not done because of the driver's inability to shut
down.
The driver for the adapter is not responding to the request to shut down. Check the version of the
driver to confirm that it supports hot plug actions.
Could not perform action because of system failure for Slot #. See system
error log for more information.
When this message is displayed, it means that some type of system error has occurred, but it
cannot be identified in detail. This message may originate from an adapter driver, the
Configuration Manager, or the System Bus Driver. Check the main console for messages and
whether the Configuration Manager, System Bus Driver, and adapter drivers are loaded. Lastly,
check the system event log for error messages.
Configuration Manager is not responding to the requested action for Slot #.
Check to see if Configuration Manager is loaded and try again.
This message occurs when the Configuration Manager is not responding to requests made by the
Configuration Manager console. Check the main console for messages and whether the
Configuration Manager is still loaded. If it is not, reload the Configuration Manager from the
main console.
ECG026/1097
PCI Hot Plug Expand Feature on Compaq Servers in Novell NetWare Environments22
System Bus Driver not responding for Slot #. Check to make sure the
System Bus Driver is loaded.
This message occurs when the System Bus Driver is not responding to requests made by the
Configuration Manager. Check the main console for messages and whether the hot plug driver is
still loaded. If not, reload the System Bus Driver from the main console.
Add/Remove performed on unknown slot (#). Try command again.
This error message occurs when the Configuration Manager console receives a message from the
Configuration Manager that refers to a slot number not recognized by the console. Try the
command again and if the error persists, unload the console and reload it, or unload and reload
the Configuration Manager.
Adding an Adapter Error Messages
The following messages may appear when you hot plug expand an adapter:
Slot # could not be initialized because there were insufficient configuration
resources.
In order for hot plug expand to successfully complete , there must be enough resources available
for the System Bus Driver to completely configure the new adapter. If the necessary resources
are not available, the adapter will be left in the powered-down state. In order to hot-plug expand
the adapter, you will need to remove other adapters to make sufficient resources available or
power cycle the system.
Slot # could not be initialized because configuration of the adapter card
type is not supported.
Some systems may not support adding PCI video adapters or adapters with non-standard PCI
configuration requirements.
Slot # cannot be initialized because a slot power fault occurred.
The system detected a power fault while adding the adapter. This usually indicates an adapter that
does not meet the power-up requirements for PCI Hot Plug.
Slot # cannot be initialized because the switch is open. Close switch and
try the ADD or REPLACE again.
The System Bus Driver has detected that the lever for the slot has not been completely closed.
Power cannot be applied to the slot until the lever is closed. Check the lever to make sure it is
closed, and try the requested action again.
ECG026/1097
No adapter is detected for Slot #. The requested action cannot continue.
A command has been issued on a slot that has no adapter. If an adapter was in the slot originally,
then an identical adapter can be inserted in the slot and power applied to the adapter. If no
adapter was in the slot when the system was powered on, then no adapter will work in the slot.
PCI Hot Plug Expand Feature on Compaq Servers in Novell NetWare Environments23
Could not perform action because of system failure for Slot #. See system
error log for more information.
When this message occurs, it means that some type of system error has occurred, but it cannot be
identified in detail. This message may originate from an adapter driver, the Configuration
Manager, or the System Bus Driver. Check the main console for messages, whether the
Configuration Manager, System Bus Driver, and adapter drivers are loaded. Lastly, check the
system event log for error messages.
Configuration Manager is not responding to the requested action for Slot #.
Check to see if Configuration Manager is loaded and try again.
This message occurs when the Configuration Manager is not responding to requests made by the
Configuration Manager console. Check the main console for messages and whether the
Configuration Manager is still loaded. If not, reload the Configuration Manager from the main
console.
System Bus Driver not responding for Slot #. Check to make sure the
System Bus Driver is loaded.
This message occurs when the System Bus Driver is not responding to requests made by the
Configuration Manager. Check the main console for messages and whether the System Bus
Driver is still loaded. If not, reload the System Bus Driver from the main console.
Add/Remove performed on unknown Slot #. Try command again.
This error message occurs when the Configuration Manager console receives a message from the
Configuration Manager that refers to a slot number unrecognized by the console. Try the
command again, and if the error persists, unload the console and reload it or unload and reload
the Configuration Manager.
Replacing an Adapter Error Messages
The following messages may appear when you replace an adapter:
Adapter is not the same as adapter being replaced in Slot #. Hot Plug
Replace cannot continue.
The REPLACE option requires that the adapter used for replacement be identical (hardware and
low-level software) to the adapter being replaced. If the Configuration Manager or the System
Bus Driver detects any differences, the replacement is not allowed.
Adapter in Slot # is currently functioning. Add/Replace is ignored.
This message occurs when REPLACE is selected for a slot that currently has an ACTIVE adapter
in it. If you wish to replace the adapter, it must first be REMOVED.
ECG026/1097
PCI Hot Plug Expand Feature on Compaq Servers in Novell NetWare Environments24
A hardware failure was detected in the adapter in Slot # and REPLACE
cannot continue.
This message occurs when the adapter driver, which is still loaded—but in a suspended state—
detects a hardware failure in the adapter and cannot initialize it. The adapter cannot be used and
needs to be replaced with another adapter.
The adapter was successfully replaced or added for Slot #. Please load the
driver from the Main Console at this time.
The adapter was successfully added (the slot has been powered on and configured) so that the
driver(s) for this adapter can be loaded. The driver must be loaded outside the Configuration
Manager console, either via the main console or by using the Install program.
ECG026/1097
PCI Hot Plug Expand Feature on Compaq Servers in Novell NetWare Environments25
Glossary
CDMUnder NWPA, device drivers are developed in two sections. Custom Device
Module is the portion that addresses interfacing to the device itself. The second
portion of the drive is the HAM, or Host Adapter Module
DDFSDevice Drive Functional Specification, the original storage driver specification
for Novell’s operating systems. The NWPA replaces it.
Disk ArrayDisk reliability and performance can be increased by grouping disks into a Disk
Array. This array is treated as a single logical device. Information is stored on the
individual disk in a variety of ways that will provide redundancy and high access.
EISAExtended Industry Standard Adapter, one of the bus standards for the PC.
Fibre ChannelSpecification for a class of high-reliability, distributed storage system. The name
comes from the fiber that may be used for communication.
HAMHost Adapter Module, the portion of the NWPA driver architecture that
interfaces with the adapter.
IDEIntegrated Dive Electronics, a standard disk interface specification. IDE also
refers to a class of CD devices, also known as ATAPI or ATA.
ISAIndustry Standard Adapter, the original bus on the AT. The original PC and XT
bus is distinguished as the 8-bit ISA.
LEDLight Emitting Diode, a solid state device that emits a colored light.
NICNetwork Interface Card.
NLMNetWare Loadable Module, the software modules dynamically loadable by the
Novell OS.
NSSDNovell Support Software Diskettes.
NWPANetWare Peripheral Architecture, the new functional specification for storage
device drivers; it replaces the DDFS.
ODIOpen Data-Link Interface, architecture for LAN drivers and protocols.
PCIPeripheral Component Interconnect (PCI) Local Bus Specification standard, a
high-performance bus standard on the PC.
PCI Hot PlugA new architecture technology that provides for the addition and replacement of
adapters and devices online, while the system is still running. There is no need to
down and power off a server to perform these replacements.
SCSISmall Computer Systems Interface, a bus standard used for primarily for disks
and CD-ROMs on the PC.
SFTSystem Fault Tolerant, technology that refers to the redundancy and reliability
that can come from multiple redundant servers.
ECG026/1097
SMPSymmetric Multiprocessor. The ability to place multiple processors in the same
box and share the processing load among the array of CPU’s.
Loading...
+ hidden pages
You need points to download manuals.
1 point = 1 manual.
You can buy points or you can get point for every manual you upload.