Sun Microsystems Sun Enterprise 10000 User's Guide

Sun Microsystems, Inc.

901 San Antonio Road

Palo Alto, CA 94303

U.S.A. 650-960-1300

Sun™ Enterprise™ 10000 Dynamic
Reconfiguration User’s Guide

Solaris™ 2.6 Hardware: 5/98

Part No. 805-5442-10

May 1998, Revision A

Send comments about this document to: [e-mail]

Copyright 1998 Sun Microsystems,Inc., 901 San Antonio Road, Palo Alto, California 94303 U.S.A. All rights reserved.
This productor document is protected by copyright and distributed under licenses restricting its use, copying, distribution, and

decompilation. No part of this product or document may be reproduced in any form by any means without prior written authorization
of Sun and its licensors, if any.Third-party software, including font technology,is copyrighted and licensed fromSun suppliers.

Parts of the productmay be derived from Berkeley BSD systems, licensed from the University of California. UNIX is a registered
trademark in the U.S. and other countries, exclusively licensed throughX/Open Company,Ltd.

Sun, Sun Microsystems,the Sun logo, SunSoft, SunDocs, SunExpress, Solaris, Solstice, DiskSuite, SunFastEthernet, Ultra Enterprise,
Sun Enterprise, AnwserBook, and OpenBoot aretrademarks, registered trademarks, or service marks of Sun Microsystems, Inc. in the
U.S. and other countries. All SPARC trademarks areused under license and are trademarks or registered trademarks of SPARC
International, Inc. in the U.S. and other countries. Productsbearing SPARCtrademarks are based upon an architecture developed by
Sun Microsystems,Inc.

The OPEN LOOK and Sun™ Graphical User Interface was developed by Sun Microsystems,Inc. for its users and licensees. Sun
acknowledges the pioneering effortsof Xerox in researching and developing the concept of visual or graphical user interfaces for the
computerindustry.Sun holds a non-exclusive license fromXerox to the Xerox Graphical User Interface, which license also covers Sun’s
licensees who implement OPEN LOOK GUIs and otherwise comply with Sun’s written license agreements.

RESTRICTEDRIGHTS: Use, duplication, or disclosureby the U.S. Government is subject to restrictions of FAR52.227-14(g)(2)(6/87)
and FAR52.227-19(6/87), or DFAR252.227-7015(b)(6/95) and DFAR227.7202-3(a).

DOCUMENTATION IS PROVIDED “AS IS” AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND
WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE
OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE
LEGALLY INVALID.

Copyright 1998 Sun Microsystems,Inc., 901 San Antonio Road, Palo Alto, Californie 94303 Etats-Unis. Tousdroits réservés.
Ce produitou document est protégé par un copyright et distribué avec des licences qui en restreignent l’utilisation, la copie, la

distribution, et la décompilation. Aucune partie de ce produitou document ne peut être reproduite sous aucune forme, par quelque
moyen que ce soit, sans l’autorisation préalable et écrite de Sun et de ses bailleurs de licence, s’il y en a. Le logiciel détenu par des tiers,
et qui comprendla technologie relative aux polices de caractères, est protégé par un copyright et licencié par des fournisseurs de Sun.

Des parties de ce produitpourront être dérivées des systèmes Berkeley BSD licenciés par l’Université de Californie. UNIX est une
marquedéposée aux Etats-Unis et dans d’autres pays et licenciée exclusivement par X/Open Company,Ltd.

Sun,Sun Microsystems, le logo Sun, SunSoft, SunDocs, SunExpress,Solaris, Solstice, DiskSuite, SunFastEthernet, Ultra Enterprise, Sun
Enterprise, AnwserBook, and OpenBoot sont des marquesde fabrique ou des marques déposées, ou marques de service, de Sun
Microsystems,Inc. aux Etats-Unis et dans d’autres pays. Toutesles marques SPARCsont utilisées sous licence et sont des marques de
fabrique ou des marques déposées de SPARCInternational, Inc. aux Etats-Unis et dans d’autres pays. Les produitsportant les marques
SPARC sont basés sur une architecturedéveloppée par Sun Microsystems, Inc.

L’interfaced’utilisationgraphique OPEN LOOK et Sun™ a été développée par Sun Microsystems, Inc. pour ses utilisateurs et licenciés.
Sun reconnaîtles efforts de pionniers de Xerox pour la rechercheet le développement du concept des interfaces d’utilisation visuelle
ou graphique pour l’industrie de l’informatique. Sun détient une licence non exclusive de Xeroxsur l’interface d’utilisation graphique
Xerox,cette licence couvrant également les licenciés de Sun qui mettent en place l’interface d’utilisation graphique OPEN LOOK et qui
en outrese conforment aux licences écrites de Sun.

CETTE PUBLICATIONEST FOURNIE "EN L’ETAT" ET AUCUNE GARANTIE, EXPRESSE OU IMPLICITE, N’EST ACCORDEE, Y COMPRIS
DES GARANTIES CONCERNANT LA VALEURMARCHANDE, L’APTITUDE DE LA PUBLICATIONA REPONDRE A UNE UTILISATION
PARTICULIERE, OU LE FAIT QU’ELLE NE SOIT PAS CONTREFAISANTE DE PRODUIT DE TIERS. CE DENI DE GARANTIE NE
S’APPLIQUERAIT PAS, DANS LA MESURE OU IL SERAIT TENU JURIDIQUEMENT NUL ET NON AVENU.

Please

Recycle

Table of Contents

Preface ix

1. Introduction to DR 1

2. DR Configuration Issues 3

Memory: dr-max-mem 3

dr-max-mem With Solaris 2.6 3
dr-max-mem With Solaris 2.5.1 4

▼ To Set dr-max-mem With Solaris 2.5.1 4

Configuration for DR Detach 6

I/O Devices 6
Memory 7

Pageable and Nonpageable Memory 7
Target Memory Constraints 8
Correctable Memory Errors 8
Swap Space 9

Reconfiguration After a DR Operation 9

When to Reconfigure 10
Disk Devices 11

DR and AP Interaction 11

iii

RPC Time-Out or Loss of Connection 12
System Quiescence Operation 13

Suspend-Safe/Suspend-Unsafe Devices 14
Special Handling for Tape Devices 14
Special Handling of Sun™ StorEdge™ A3000 15

DR and DDI 15

DR and DDI_DETACH 15
DR and DDI_SUSPEND/DDI_RESUME 16
Adding Suspend-Safe Devices 17
Adding Suspend-Bypass Devices 18
DR Detach-Safe Devices 19

3. Using Dynamic Reconfiguration 21

Attaching a System Board 21

Init Attach 21
Complete Attach 22
Attach Buttons 23

▼ To Attach a Board With Hostview 23
▼ To Attach a Board By Using dr(1M) 27

Detaching a System Board 30

Drain 30
Complete Detach 31

Network Devices 31
Non-Network Devices 32
Processors 33
Finishing the Complete Detach Operation 33

Hostview Detach Buttons 34

▼ To Detach a Board With Hostview 35

iv Sun Enterprise 10000 Dynamic Reconfiguration User’s Guide • May 1998

▼ To Detach a Board By Using dr(1M) 38

Viewing Domain Information 41

▼ To View Domain Information with Hostview 42
▼ To Specify How Windows Are Updated 42
▼ To View DR CPU Configuration Information 43
▼ To View DR Memory Configuration Information 44
▼ To View DR Device Configuration Information 47
▼ To View DR Device Detailed Information 48
▼ To View DR OBP Configuration Information 49
▼ To View the Suspend-Unsafe Devices Across the Entire Domain 50

Index 1

Table of Contents v

vi Sun Enterprise 10000 Dynamic Reconfiguration User’s Guide • May 1998

Figures

FIGURE 3-1 Attach—Board and Domain Selection Window 24
FIGURE 3-2 Dynamic Reconfiguration Window With init attach Button 25
FIGURE 3-3 Dynamic Reconfiguration Window 26
FIGURE 3-4 Detach—Board and Domain Selection Window 36
FIGURE 3-5 Dynamic Reconfiguration Window for drain Operation 37
FIGURE 3-6 System Information Buttons 42
FIGURE 3-7 DR Properties Window 42
FIGURE 3-8 DR CPU Configuration Window 43
FIGURE 3-9 DR Memory Configuration Window 45
FIGURE 3-10 DR Device Configuration Window 47
FIGURE 3-11 DR Detail Device Window 48
FIGURE 3-12 DR OBP Configuration Window 50
FIGURE 3-13 DR Unsafe Devices Window 51

vii

viii Sun Enterprise 10000 Dynamic Reconfiguration User’s Guide • May 1998

Preface

This book describes the Dynamic Reconfiguration (DR) feature, which enables you
to logically attach and detach system boards from the Sun™ Enterprise™ 10000
server while other domains continue running.

Before You Read This Book

This book is intended for the Sun Enterprise 10000 system administrator. Users of
the Enterprise 10000 system should have a working knowledge of UNIX
particularly those based on the Solaris™ operating environment. If you do not have
such knowledge, first read the Solaris User and System Administrator in
AnswerBook™ format provided with this system and consider UNIX system
administration training.

®

systems,

How This Book Is Organized

This document contains the following chapters:
Chapter 1 “Introduction to DR” introduces basic concepts related to the Dynamic

Reconfiguration feature.
Chapter 2 “DR Configuration Issues” describes how to configure the Dynamic

Reconfiguration system before you begin using it.
Chapter 3 “Using Dynamic Reconfiguration” describes how to use DR to attach and

detach system boards.

ix

Using UNIX Commands

This document does not contain information on basic UNIX commands and
procedures such as shutting down the system, booting the system, and configuring
devices.

See one or more of the following sources for this information:

■ AnswerBook online documentation for the Solaris 2.x software environment,

particularly those dealing with Solaris system administration

■ Other software documentation that you received with your system

Typographic Conventions

The following table describes the typographic changes used in this book.

TABLEP-1 Typographic Conventions

Typeface or
Symbol Meaning Examples

AaBbCc123 The names of commands, files,

and directories; on-screen
computer output.

AaBbCc123

AaBbCc123 Book titles, new words or terms,

x Sun Enterprise 10000 Dynamic Reconfiguration User’s Guide • May 1998

What you type, when
contrasted with on-screen
computer output.

words to be emphasized.
Command-line variable; replace
with a real name or value.

Edit your .login file.
Use ls -a to list all files.

% You have mail.
% su

Password:

Read Chapter 6 in the User ’s Guide.
These are called class options.
You must be root to do this.
To delete a file, type rm filename.

Shell Prompts

The following table shows the default system prompt and superuser prompt for the
C shell, Bourne shell, and Korn shell.

TABLEP-2 Shell Prompts

Shell Prompt

C shell machine_name%
C shell superuser machine_name#
Bourne shell and Korn shell $
Bourne shell and Korn shell superuser #

Related Documentation

DR is normally started from the Hostview GUI in the SSP environment. See the
following documentation for more information about DR:

■ SMCC Release Notes Supplement , a printed document in your media box, part

number 805-3537-10. This document may contain Dynamic Reconfiguration
Release Notes.

■ Sun Enterprise 10000 SSP User’s Guide, part number 805-2955-10

■ Sun Enterprise 10000 SSP Reference Manual, part number 805-3362-10

xi

Ordering Sun Documents

SunDocsSMis a distribution program for Sun Microsystems technical documentation.
Contact SunExpress for easy ordering and quick delivery. You can find a listing of
available Sun documentation on the World Wide Web.

TABLEP-3 SunExpress Contact Information

Country Telephone Fax

Belgium 02-720-09-09 02-725-88-50
Canada 1-800-873-7869 1-800-944-0661
France 0800-90-61-57 0800-90-61-58
Germany 01-30-81-61-91 01-30-81-61-92
Holland 06-022-34-45 06-022-34-46
Japan 0120-33-9096 0120-33-9097
Luxembourg 32-2-720-09-09 32-2-725-88-50
Sweden 020-79-57-26 020-79-57-27
Switzerland 0800-55-19-26 0800-55-19-27
United Kingdom 0800-89-88-88 0800-89-88-87
United States 1-800-873-7869 1-800-944-0661

World Wide Web: http://www.sun.com/sunexpress/

Sun Documentation on the Web

The docs.sun.com web site enables you to access Sun technical documentation on the
World Wide Web. You can browse the docs.sun.com archive or search for a specific
book title or subject at http://docs.sun.com.

xii Sun Enterprise 10000 Dynamic Reconfiguration User’s Guide • May 1998

Sun Welcomes Your Comments

We are interested in improving our documentation and welcome your comments
and suggestions. You can email your comments to us at [e-mail].
Please include the part number of your document in the subject line of your email.

xiii

xiv Sun Enterprise 10000 Dynamic Reconfiguration User’s Guide • May 1998

CHAPTER

1

Introduction to DR

Dynamic Reconfiguration (DR) enables you to logically attach and detach system
boards to and from the operating system without causing machine downtime. DR is
used in conjunction with hot swap, which is the process of physically removing or
inserting a system board. You can use DR to add a new system board, reinstall a
repaired system board, or modify the domain configuration on the Sun Enterprise
10000 system.

If a system board is being used by the operating system, you must detach it before
you can power it off and remove it. After a new or upgraded system board is
inserted and powered on, you may attach it to the operating system.

You can execute DR operations through the Hostview GUI (see hostview(1M))or
through the dr(1M) shell application. DR supports the following operations:

■ DR Attach – Logically attaches a system board to the operating system running in

a domain. A system board is logically attached when its resources—processors,
memory, and I/O adapters—are configured into a domain and are available to
the Solaris operating system. The system board must already be present in the
system, powered on, and not be a member of a domain. Normally, you attach a
system board after it is inserted and powered on by your service provider or after
it is detached from another domain.

■ DR Detach – Logically detaches a system board from the operating system. A

system board is logically detached when its resources—processors, memory, and
I/O adapters—are removed from the domain configuration and are no longer
available to the domain. Normally, you detach a system board to either move it to
another domain or prepare it for removal.

While DR operations are being performed within a domain, dr_daemon(1M) (see
the Solaris Reference Manual for SMCC-Specific Software) and the operating system
write messages regarding the status or exceptions to the domains’ syslog message
buffer ( /var/adm/messages) and the SSP message files (
messages and
information displayed by Hostview and the dr(1M) shell application, the
dr_daemon(1M) and operating system messages are useful for determining the
status of DR requests.

$SSPOPT/adm/messages). In addition to the status and exception

$SSPOPT/adm/host/

1

Note – Only one DR operation per platform can be active at any time. A DR

operation that is partially completed and then dismissed within one domain does
not prevent a subsequent DR operation from being started in a different domain. A
partially completed DR operation must be finished before a subsequent DR
operation is permitted in the same domain.

2 Sun Enterprise 10000 Dynamic Reconfiguration User’s Guide • May 1998

CHAPTER

2

DR Configuration Issues

This chapter describes how to configure a domain for all DR operations and
capabilities. The DR features are enabled only when the OpenBoot™ PROM (OBP)
environment variable dr-max-mem is set to a non-zero value. The sections in this
chapter include more information about dr-max-mem.

Note – DR features are disabled on domains that have less than 512-Mbytes of

memory.

Caution – Be careful when choosing the slot into which a board is inserted to

prevent disk controller renumbering. For more information, see “Reconfiguration
After a DR Operation” on page 9.

Memory: dr-max-mem

The value for dr-max-mem depends on the version of the Solaris operating
environment (2.5.1, 2.6, or higher) that is running in the domain. This section
includes information on both versions.

dr-max-mem With Solaris 2.6

With Solaris 2.6 or higher, the memory-related data structures are dynamically
allocated during the DR Attach operation. They are also dynamically removed
during the DR Detach operation; therefore, dr-max-mem with Solaris 2.6 becomes an
on/off switch. With dr-max-mem set to zero, DR operations are disabled. This is
true no matter which version of Solaris is running in the domain.

3

dr-max-mem With Solaris 2.5.1

The kernel has a number of memory-related data structures such as page structures,
which are statically allocated at boot time and are based on the amount of physical
memory in the domain at that time. Use DR Attach to dynamically add a board and
its physical memory after the domain is booted. This extra memory can be
supported by the kernel only if enough memory data structures are allocated at boot
time to support it. These structures cannot be added dynamically after boot time.

To reserve enough memory data structures to support DR Attach operations, each
domain supports the OBP environment variable, dr-max-mem, which the kernel
reads at boot time. dr-max-mem specifies the maximum number of megabytes to
which the domain can grow without requiring a reboot. Each domain has its own
unique copy of dr-max-mem.

▼ To Set dr-max-mem With Solaris 2.5.1

1. Calculate the optimum value for dr-max-mem by combining the amount of
memory most likely to be added during all DR Attaches and the current amount
of memory present in the domain and setting dr-max-mem to the total.

Note that if dr-max-mem is too large relative to the memory in the domain, its size
can impact the performance of the operating system. Therefore, the operating system
limits the maximum value of dr-max-mem at boot time, as follows:

TABLE2-1 dr-max-mem Maximum Values

Current Physical
Memory

256-Mbytes 0
512-Mbytes 16-Gbytes
1024-Mbytes 32-Gbytes
2048-Mbytes 64-Gbytes

dr-max-mem

Maximum Value

If the value of dr-max-mem is smaller than the amount of physical memory present
when the domain is booted, the operating system sets its working copy of dr-max-
mem to the current memory size. You cannot attach more memory, but you can
detach, then re-attach memory. The maximum amount of memory you can re-attach
in this manner is the amount present when the domain was booted. Note that the
OBP variable dr-max-mem is not modified in this situation.

Caution – Set dr-max-mem high enough so that all anticipated new memory can be

dynamically attached, but no higher. If you set it too low and later attach a board
whose memory combined with domain memory exceeds the value of dr-max-mem,

4 Sun Enterprise 10000 Dynamic Reconfiguration User’s Guide • May 1998

the memory on that board will not be attached. If you set the value of dr-max-mem
too high, you over-allocate data structures, which can waste available memory and
adversely affect system performance. If you set it to zero, the DR functions are
disabled.

dr-max-mem must be set before the domain is booted.

2. Set the dr-max-mem environment variable by bringing up the OBP prompt for the
domain and typing the following command:

ok# setenv-dr dr-max-mem

NNN

where NNN is the number of megabytes of memory to be supported by the domain
after the boards are attached. The value of dr-max-mem persists across domain
reboots, and is only applicable to that particular domain. This value will apply to all
boot environments for the domain.

Note – Once you have set the dr-max-mem value for a domain, that value remains

the same no matter which Solaris boot disk you select.

If the dr-max-mem variable is non-zero, the following messages are displayed at
boot time in the domain’s syslog message buffer (/var/adm/messages):

DR: current memory size is
DR: capacity to allow an additional

XXX

MBytes

YYY

MBytes of memory

In this message, XXX represents the amount of physical memory available to the
operating system and is effectively the same as the operating system variable,
physinstalled. YYY is the difference between dr-max-mem and XXX.

When a board with memory is successfully attached or detached, another message is
displayed:

DR: capacity to allow an additional

ZZZ

MBytes of memory

In this message, ZZZ represents the updated amount of memory that can still be
attached.

Chapter 2 DR Configuration Issues 5

Configuration for DR Detach

This section describes how to configure DR before you perform a detach operation.

Note – The DR Detach feature requires that the OBP variable dr-max-mem is set to

a non-zero value. This setting is required at the time the domain is booted.

I/O Devices

The DR Detach feature relies on the Alternate Pathing (AP) feature or Solstice™
DiskSuite™ mirroring when used to detach a board that hosts I/O controllers that
are attached to vital system resources. Currently, AP and Solstice DiskSuite do not
work together; however, AP does work with Veritas Volume Manager. If, for
example, the root or /usr partition is on a disk attached to a controller on the board,
the board cannot be detached unless there is a hardware alternate path to the disk—
and AP has been configured to take advantage of it—or the disk is mirrored. The
alternate path or the mirrors must be hosted by other boards in the system. The
same applies to network controllers. The board that hosts the Ethernet controller that
connects the SSP to the Enterprise 10000 platform cannot be detached unless an
alternate path exists to an Ethernet controller on another board for this network
connection.

The domain swap space should be configured as multiple partitions on disks
attached to controllers hosted by different boards. With this kind of configuration, a
particular swap partition is not a vital resource because swap partitions can be
added and deleted dynamically (see swap(1M) for more information).

Note – When memory (swapfs) or disk swap space is detached, there must be

enough memory or swap disk space remaining in the domain to accommodate
currently running programs.

A board that is hosting non-vital system resources can be detached whether or not
there are alternate paths to the resources. All of the board's devices must be closed
before the board can be detached; all of its file systems must be unmounted; and, its
swap partitions must be deleted. You may have to kill processes that have open files
or devices, or place a hard lock on the file systems (using lockfs(1M)) before you
unmount the boards. There is a domain disruption penalty associated with the
detach operation.

6 Sun Enterprise 10000 Dynamic Reconfiguration User’s Guide • May 1998

All I/O device drivers involved with I/O devices on the board(s) must support the
DDI_DETACH option in the driver detach entry point. This option releases all system
resources associated with that device or adapter.

Memory

If you use memory interleaving between system boards, those system boards cannot
be detached because DR does not yet support interboard interleaving. By default,
hpost(1M) does not set up boards with interleaved memory. Look for the following
line in the hpost(1M) file .postrc (see postrc(4)):

mem_board_interleave_ok

If mem_board_interleave_ok is present, you may not be able to detach a board
that contains memory.

Pageable and Nonpageable Memory

Before you can detach a board, the operating system must vacate the memory on
that board. Vacating a board means flushing its pageable memory to swap space and
copying its permanent memor y (that is, nonpageable kernel and OBP memory) to
another memory board. To relocate nonpageable memory, the operating system on a
domain must be temporarily suspended, or quiesced. The length of the suspension
depends on the domain I/O configuration and the current running workload.
Detaching a board with nonpageable memory is the only time when the operating
system is suspended; therefore, you should know where nonpageable memory
resides, so you can avoid significantly impacting the domain’s operation. When
permanent memory is on the board, the operating system must find other memory
to receive the copy.

You can use the dr(1M) command drshow(1M) to determine if a board’s memory is
pageable or nonpageable:

% dr
dr> drshow board_number mem

Similarly, you can determine if a board’s memory is pageable by looking at the DR
Memory Configuration window, which is available when you perform a detach
operation within Hostview. The DR Memory Configuration window is described in
“Viewing Domain Information” on page 41.

Chapter 2 DR Configuration Issues 7