Fujitsu SPARC Enterprise M4000, SPARC Enterprise M5000, SPARC Enterprise M8000, SPARC Enterprise M9000 Features Manual

SPARC Enterprise M4000/M5000/M8000/M9000 Servers

Dynamic Reconfiguration (DR) User's Guide

Part No.: E27809-01, Manual Code: C120-E335-09EN January 2012

Copyright © 2007, 2012, Fujitsu Limited. All rights reserved. Oracle and/or its affiliates provided technical input and review on portions of this material. Oracle and/or its affiliates and Fujitsu Limited each own or control intellectual property rights relating to products and technolog y described in this

document, and such products, technology and this document are protected by copyright laws, patents, and other intellectual property laws and international treaties.

This document and the product and technology to which it pertains are distributed under licenses restricting their use, copying, distribution, and decompilation. No part of such product or technology, or of this document, may be reproduced in any form by any means without prior written authorization of Oracle and/or its affiliates and Fujitsu Limited, and their applicable licensors, if any. The furnishings of this document to you does not give you any rights or licenses, express or implied, with respect to the product or technology to which it pertains, and this document does not contain or represent any commitment of any kind on the part of Oracle or Fujitsu Limited, or any affiliate of either of them.

This document and the product and technology described in this document may incorporate third-party intellectual property copyrighted by and/or licensed from the suppliers to Oracle and/or its affiliates and Fujitsu Limited, including software and font technology.

Per the terms of the GPL or LGPL, a copy of the source code governed by the GPL or LGPL, as applicab le, is ava ilable upon request by the End User. Please contact Oracle and/or its affiliates or Fujitsu Limited.

This distribution may include materials developed by third parties. Parts of the product may be derived from Berkeley BSD systems, licensed from the University of California. UNIX is a registered trademark in the U.S. and

in other countries, exclusively licensed through X/Open Company, Ltd. Oracle and Java are registered trademarks of Oracle and/or its affiliates. Fujitsu and the Fujitsu logo are registered trademarks of Fujitsu Limited. All SPARC trademarks are used under license and are registered trademarks of SPARC International, Inc. in the U.S. and other countries. Products bearing

SPARC trademarks are based upon architectures developed by Oracle and/or its affiliates. SPARC64 is a trademark of SPARC International, Inc., used under license by Fujitsu Microelectronics, Inc. and Fujitsu Limited. Other names may be trademarks of their respective owners.

United States Government Rights - Commercial use. U.S. Government users are subject to the standard government user license agreements of Oracle and/or its affiliates and Fujitsu Limited and the applicable provisions of the FAR and its supplements.

Disclaimer: The only warranties granted by Oracle and Fujitsu Limited, and/or any affiliate of either of them in connection with this document or any product or technology described herein are those expressly set forth in the license agreement pursuant to which the product or technology is provided. EXCEPT AS EXPRESSLY SET FORTH IN SUCH AGREEMENT, ORACLE OR FUJITSU LIMITED, AND/OR THEIR AFFILIATES MAKE NO REPRESENTATIONS OR WARRANTIES OF ANY KIND (EXPRESS OR IMPLIED) REGARDING SUCH PRODUCT OR TECHNOLOGY OR THIS DOCUMENT, WHICH ARE ALL PROVIDED AS IS, AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING WITHOUT LIMITATION ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NONINFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID. Unless otherwise expressly set forth in such agreement, to the extent allowed by applicable law, in no event shall Oracle or Fujitsu Limited, and/or any of their affiliates have any liability to any third party under any legal theory for any loss of revenues or profits, loss of use or data, or business interruptions, or for any indirect, special, incidental or consequential damages, even if advised of the possibility of such damages.

DOCUMENTATION IS PROVIDED “AS IS” AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.

Please

Recycle

Copyright © 2007, 2012, Fujitsu Limited. Tous droits réservés. Oracle et/ou ses sociétés affiliées ont fourni et vérifié des donn ées techniq ues de certaines parties de ce composant. Oracle et/ou ses sociétés affiliées et Fujitsu Limited détiennent et contrôlent chacune des droits de propriété intellectuelle relatifs aux produits et

technologies décrits dans ce document. De même, ces produits, technologies et ce document sont protégés par des lois sur le copyright, des brevets, d’autres lois sur la propriété intellectuelle et des traités internationaux.

Ce document, le produit et les technologies afférents sont exclusivement distribués avec des licences qui en restreignent l’utilisation, la copie, la distribution et la décompilation. Aucune partie de ce produit, de ces technologies ou de ce document ne peut être reproduite sous quelque forme que ce soit, par quelque moyen que ce soit, sans l’autorisation écrite préalable d’Oracle et/ou ses sociétés affiliées et de Fujitsu Limited, et de leurs éventuels bailleurs de licence. Ce document, bien qu’il vous ait été fourni, ne vous confère aucun droit et aucune licence, expresses ou tacites, concernant le produit ou la technologie auxquels il se rapporte. Par ailleurs, il ne contient ni ne représente aucun engagement, de quelque type que ce soit, de la part d’Oracle ou de Fujitsu Limited, ou des sociétés affiliées de l’une ou l’autre entité.

Ce document, ainsi que les produits et technologies qu’il décrit, peuvent inclure des droits de propriété intellectuelle de parties tierces protégés par copyright et/ou cédés sous licence par des fournisseurs à Oracle et/ou ses sociétés affiliées et Fujitsu Limited, y compris des logiciels et des technologies relatives aux polices de caractères.

Conformément aux conditions de la licence GPL ou LGPL, une copie du code source régi par la licence GPL ou LGPL, selon le cas, est disponible sur demande par l’Utilisateur final. Veuillez contacter Oracle et/ou ses sociétés affiliées ou Fujitsu Limited.

Cette distribution peut comprendre des composants développés par des parties tierces. Des parties de ce produit peuvent être dérivées des systèmes Berkeley BSD, distribués sous licence par l’Université de Californie. UNIX est une marque

déposée aux États-Unis et dans d’autres pays, distribuée exclusivement sous licence par X/Open Company, Ltd. Oracle et Java sont des marques déposées d’Oracle Corporation et/ou de ses sociétés affiliées. Fujitsu et le logo Fujitsu sont des marques déposées de

Fujitsu Limited. Toutes les marques SPARC sont utilisées sous licence et sont des marques déposées de SPARC International, Inc., aux États-Unis et dans d’autres pays. Les

produits portant la marque SPARC reposent sur des architectures développées par Oracle et/ou ses sociétés affiliées. SPARC64 est une marque de SPARC International, Inc., utilisée sous licence par Fujitsu Microelectronics, Inc. et Fujitsu Limited. Tout autre nom mentionné peut corresp ondre à des marq ues appartenant à d’autres propriétaires.

Avis de non-responsabilité : les seules garanties octroyées par Oracle et Fujitsu Limited et/ou toute société affiliée de l’une ou l’autre entité en rapport avec ce document ou tout produit ou toute technologie décrits dans les présentes correspondent aux garanties expressément stipulées dans le contrat de licence régissant le produit ou la technologie fournis. SAUF MENTION CONTRAIRE EXPRESSÉMENT STIPULÉE DANS CE CONTRAT, ORACLE OU FUJITSU LIMITED ET LES SOCIÉTÉS AFFILIÉES À L’UNE OU L’AUTRE ENTITÉ REJETTENT TOUTE REPRÉSENTATION OU TOUTE GARANTIE, QUELLE QU’EN SOIT LA NATURE (EXPRESSE OU IMPLICITE) CONCERNANT CE PRODUIT, CETTE TECHNOLOGIE OU CE DOCUMENT, LESQUELS SONT FOURNIS EN L’ÉTAT. EN OUTRE, TOUTE S LES CONDITIONS, REPRÉSENTATIONS ET GARANTIES EXPRESSES OU TACITES, Y COMPRIS NOTAMMENT TOUTE GARANTIE IMPLICITE RELATIVE À LA QUALITÉ MARCHANDE, À L’APTITUDE À UNE UT ILISATION PARTICULIÈRE OU À L’ABSENCE DE CONTREFAÇON, SONT EXCLUES, DANS LA MESU RE AUTORISÉE PAR LA LOI APPLICABLE. Sauf mention contraire expressément stipulée dans ce contrat, dans la mesure autorisée par la loi applicable, en aucun cas Oracle ou Fujitsu Limited et/ou l’une ou l’autre de leurs sociétés affiliées ne sauraient être tenues responsables envers une quelconque partie tierce, sous quelque théorie juridique que ce soit, de tout manque à gagner ou de perte de profit, de problèmes d’utilisation ou de perte de données, ou d’interruptions d’activités, ou de tout dommage indirect, spécial, secondaire ou consécutif, même si ces entités ont été préalablement informées d’une telle éventualité.

LA DOCUMENTATION EST FOURNIE « EN L’ÉTAT » ET TOUTE AUTRE CONDITION, DÉCLARATION ET GARANTIE, EXPRESSE OU TACITE, EST FORMELLEMENT EXCLUE, DANS LA MESURE AUTORISÉE PAR LA LOI EN VIGUEUR, Y COMPRIS NOTAMMENT TOUTE GARANTIE IMPLICITE RELATIVE À LA QUALITÉ MARCHANDE, À L’APTITUDE À UNE UTILISATION PARTICULIÈRE OU À L’ABSENCE DE CONTREFAÇON.

Preface ix

1. Overview of Dynamic Reconfiguration 1–1

1.1 DR 1–1

1.2 Basic DR Functions 1–5

1.2.1 Adding a System Board 1–6

1.2.2 Deleting a System Board 1–6

1.2.3 Moving a System Board 1–6

1.2.4 Replacing a System Board 1–7

1.3 Security 1–7

1.4 Overview of DR User Interfaces 1–7

2. What You Must Know Before Using DR 2–1

2.1 System Configuration 2–1

2.1.1 System Board Components 2–1

2.1.1.1 CPU 2–4

2.1.1.2 Memory 2–5

2.1.1.3 I/O Device 2–9

2.1.2 System Board Configuration Requirements 2–10

2.1.3 System Board Pool Function 2–10

2.1.4 Checklists for System Configuration 2–11

2.1.5 Reservation of Domain Configuration Changes 2–12

2.2 Conditions and Settings Using XSCF 2–13

2.2.1 Conditions Using XSCF 2–13

2.2.2 Settings Using XSCF 2–13

2.2.2.1 Configuration Policy Option 2–14

2.2.2.2 Floating Board Option 2–14

2.2.2.3 Omit-memory Option 2–15

2.2.2.4 Omit-I/O Option 2–16

2.3 Conditions and Settings Using Oracle Solaris OS 2–16

2.3.1 I/O and Software Requirements 2–16

2.3.2 Settings of Kernel Cage Memory 2–17

2.3.3 Setting of Oracle Solaris Service Management Facility (SMF) 2–18

2.4 Status Management 2–18

2.4.1 Domain Status 2–18

2.4.2 System Board Status 2–19

2.4.3 Flow of DR Processing 2–21

2.4.3.1 Flowchart: Adding a System Board 2–21

2.4.3.2 Flowchart: Deleting a System Board 2–22

2.4.3.3 Flowchart: Moving a System Board 2–23

2.4.3.4 Flowchart: Replacing System Board 2–25

2.5 Operation Management 2–27

2.5.1 I/O Device Management 2–27

2.5.2 Swap Area 2–27

2.5.2.1 Swap Area at System Board Addition 2–27

2.5.2.2 Swap Area at System Board Deletion 2–27

2.5.3 Real-time Processes 2–28

2.5.4 Memory Mirror Mode 2–28

vi SPARC Enterprise Mx000 Servers Dynamic Reconfiguration User’s Guide • January 2012

2.5.5 Capacity on Demand (COD) 2–29

2.5.6 XSCF Failover 2–29

2.5.7 Kernel Memory Board Deletion 2–29

2.5.8 Deletion of Board with CD-RW/DVD-RW Drive 2–30

2.5.9 SPARC64 VII+, SPARC64 VII, and SPARC64 VI Processors and CPU Operational Modes 2–30

2.5.9.1 CPU Operational Modes 2–31

3. DR User Interface 3–1

3.1 How To Use the DR User Interface 3–1

3.1.1 Displaying Domain Information 3–2

3.1.2 Displaying Domain Status 3–5

3.1.3 Displaying System Board Information 3–7

3.1.4 Displaying Device Information 3–10

3.1.5 Displaying System Board Configuration Information 3–13

3.1.6 Adding a System Board 3–15

3.1.7 Deleting a System Board 3–17

3.1.8 Moving a System Board 3–19

3.1.9 Replacing a System Board 3–22

3.1.10 Reserving a Domain Configuration Change 3–25

3.2 Command Reference 3–26

3.3 XSCF Web 3–27

3.4 RCM Script 3–27

4. Practical Examples of DR 4–1

4.1 Flow of DR Operation 4–2

4.1.1 Flow: Adding a System Board 4–3

4.1.2 Flow: Deleting a System Board 4–4

4.1.3 Flow: Moving a System Board 4–5

4.1.4 Flow: Replacing a System Board 4–6

Contents vii

4.2 Example: Adding a System Board 4–7

4.3 Example: Deleting a System Board 4–9

4.4 Example: Moving a System Board 4–11

4.5 Examples: Replacing a System Board 4–13

4.5.1 Example: Replacing a Uni-XSB System Board 4–13

4.5.2 Example: Replacing a Quad-XSB System Board 4–16

4.6 Examples: Reserving Domain Configuration Changes 4–20

4.6.1 Example: Reserving a System Board Add 4–20

4.6.2 Example: Reserving a System Board Delete 4–22

4.6.3 Example: Reserving a System Board Move 4–23

A. Message Meaning and Handling A–1

A.1 Oracle Solaris OS Messages A–1

A.1.1 Transition Messages A–1

A.1.2 PANIC Messages A–3

A.1.3 Warning Messages A–4

A.2 Command Messages A–24

A.2.1 addboard A–24

A.2.2 deleteboard A–27

A.2.3 moveboard A–29

A.2.4 setdcl A–33

A.2.5 setupfru A–34

A.2.6 showdevices A–35

B. Example: Confirm Swap Space Size B–1

Index Index–1

viii SPARC Enterprise Mx000 Servers Dynamic Reconfiguration User’s Guide • January 2012

Preface

This guide describes the Dynamic Reconfiguration (DR) feature of SPARC Enterprise M4000/M5000/M8000/M9000 servers from Oracle and Fujitsu. DR enables users to add, remove or exchange system boards in the M4000/M5000 (midrange) and M8000/M9000 (high-end) servers while the domains that contain these boards remain up and running. The M3000 server does not support DR.

Some references to server names and document names are abbreviated for readability. For example, if you see a reference to the M9000 server, note that the full product name is the SPARC Enterprise M9000 server. And if you see a reference to the XSCF Reference Manual, note that the full document name is the SPARC Enterprise M3000/M4000/M5000/M8000/M9000 Servers XSCF Reference Manual.

Before reading this document, you should read the overview guide for your server, the SPARC Enterprise M3000/M4000/M5000/M8000/M9000 Servers Administration

Guide, and the SPARC Enterprise M3000/M4000/M5000/M8000/M9000 Servers XSCF User’s Guide.

At publication of this document, servers described herein were shipping with XCP 1110 firmware supported or installed. That might no longer be the latest available version, or the version now installed. Always see the Product Notes that apply to the firmware on your server, and those that apply to the latest firmware release.

This chapter includes the following sections:

■ “Audience” on page x

■ “Related Documentation” on page x

■ “Text Conventions” on page xii

■ “Syntax of the Command-Line Interface (CLI)” on page xii

■ “Documentation Feedback” on page xiii

Audience

This guide is written for experienced system administrators with working knowledge of computer networks and advanced knowledge of the Oracle Solaris Operating System (Oracle Solaris OS).

Text Conventions

This manual uses the following fonts and symbols to express specific types of information.

Font/Symbol Meaning Example

AaBbCc123

AaBbCc123 The names of commands, files, and

Italic Indicates the name of a reference

" " Indicates names of chapters,

What you type, when contrasted with on-screen computer output.

This font represents the example of command input in the frame.

directories; on-screen computer output.

This font represents the example of command output in the frame.

manual, a variable, or userreplaceable text.

sections, items, buttons, or menus.

XSCF> adduser jsmith

XSCF> showuser -P User Name: jsmith Privileges: useradm

auditadm

See the SPARC Enterprise M3000/M4000/M5000/M8000/M9000 Servers XSCF User’s Guide.

See Chapter 2, "System Features."

Syntax of the Command-Line Interface (CLI)

The command syntax is as follows:

■ A variable that requires input of a value must be put in Italics.

■ An optional element must be enclosed in [].

■ A group of options for an optional keyword must be enclosed in [] and delimited

by |.

xii SPARC Enterprise Mx000 Servers Dynamic Reconfiguration User’s Guide • January 2012

Documentation Feedback

If you have any comments or requests regarding this document, go to the following websites:

■ For Oracle users:

http://www.oracle.com/goto/docfeedback

Include the title and part number of your document with your feedback:

SPARC Enterprise M4000/M5000/M8000/M9000 Servers Dynamic Reconfiguration (DR) User’s Guide, part number E27809-01

■ For Fujitsu users:

http://www.fujitsu.com/global/contact/computing/sparce_index.html

Preface xiii

xiv SPARC Enterprise Mx000 Servers Dynamic Reconfiguration User’s Guide • January 2012

CHAPTER

Overview of Dynamic Reconfiguration

This chapter provides an overview of Dynamic Reconfiguration, which is controlled by the eXtended System Control Facility (XSCF).

This chapter includes these sections:

■ Section 1.1, “DR” on page 1-1

■ Section 1.2, “Basic DR Functions” on page 1-5

■ Section 1.3, “Security” on page 1-7

■ Section 1.4, “Overview of DR User Interfaces” on page 1-7

1.1 DR

Dynamic Reconfiguration (referred to as DR, in this document) enables hardware resources such as processors, memory, and I/O to be added and deleted even while the Oracle Solaris Operating System (referred to as Oracle Solaris OS in this document) is running.

DR has three basic functions; i.e., addition, deletion and move, which can be used for the following purposes.

■ Add system boards without stopping the Oracle Solaris OS of the domain, to

improve business operations or handle higher system loads.

■ Temporarily remove a faulty system board for parts replacement without

stopping the Oracle Solaris OS of the domain, in the event of an error that causes the system board to become degraded.

1-1

■ Move a resource from one domain to another while continuously operating the

System boards

Uni-XSB

MBU

XSB

Quad-XSB

MBU

XSB XSB XSB XSB XSB XSB XSB XSB

CMU

IOU

CMU

IOU

domains without physically removing or inserting a system board. Resources can be moved to balance the loads on multiple domains, or to share common I/O resources between domains.

SPARC Enterprise M4000/M5000/M8000/M9000 servers have a unique partitioning feature that can divide one physical system board (PSB) into one logical board (undivided status) or four logical boards. A PSB that is logically divided into one board (undivided status) is called a Uni-XSB, whereas a PSB that is logically divided into four boards is called a Quad-XSB. Each composition of physical unit of the divided PSB is called an eXtended System Board (XSB). These XSBs can be combined freely to create domains.

DR functions on these servers are performed on an XSB. This manual uses the term system board unless physical units of PSB and XSB are described. For an explanation of each term, see

TABLE 1-2.

Note – This document explains DR functions on system boards. Use the Oracle

Solaris command cfgadm(1M) to execute DR on I/O devices, including PCI cards. For more information, please see the Service Manual for your server, and the cfgadm(1M) and cfgadm_pci(1M) man pages.

FIGURE 1-1 Uni-XSB and Quad-XSB (Midrange Servers)

1-2 SPARC Enterprise Mx000 Servers Dynamic Reconfiguration User’s Guide • January 2012

FIGURE 1-2 Uni-XSB and Quad-XSB (High-end Servers)

Uni-XSB

XSB

Quad-XSB

XSB XSB XSB XSB

System boards

CMU

IOU

CMU

IOU

TABLE 1-1 and TABL E 1 -2 list DR-related terms.

TABLE 1-1 Basic DR Terms

Term Definition

Add To connect a system board to a domain and configure it into the

Oracle Solaris OS of the domain.

Delete To unconfigure a system board from the Oracle Solaris OS of a

Move To disconnect a system board from a domain and then connect the

Release To delete a registered system board from the DCL.

Assign To assign a system board to a domain.

Unassign To release a system board from a domain.

Connect To connect a system board to a domain.

Disconnect To disconnect a system board from a domain.

Configure To configure a system board in the Oracle Solaris OS.

domain and disconnect it from the domain.

system board to another domain.

called DCL).

Chapter 1 Overview of Dynamic Reconfiguration 1-3

TABLE 1-1 Basic DR Terms (Continued)

Term Definition

Unconfigure To unconfigure a system board in the Oracle Solaris OS.

Reserve To reserve a system board such that it is assigned to or unassigned

from a domain on the next reboot or power-cycle.

Install To insert a system board into a system.

Remove To remove a system board from a system.

Replace To remove a system board and then mount it or a new system board,

for system maintenance and inspection.

TABLE 1-2 Terms Related to Hardware Configurations

Term Definition

CPU/Memory board unit (CMU)

Motherboard Unit (MBU)

Unit equipped with a CPU module, and memory. High-end servers only.

Unit for midrange servers. A CMU is mounted on this board. Midrange servers only.

I/O unit (IOU) Unit equipped with a PCI card and a disk drive unit.

Physical System Board (PSB)

The PSB is made up of physical parts, and can include 1 CMU and 1 IOU or just 1 CMU. In midrange servers, the CMU is mounted on a MBU. A PSB also can be used to describe a physical unit for addition/deletion/exchange of hardware. The PSB can be used in one of two methods, one complete unit (undivided status) or divided into four subunits.

eXtended System Board (XSB)

The XSB is made of physical parts. In the XSB, the PSB can be either one complete unit (undivided status) or divided into four subunits. The XSB is a unit used for domain construction and identification, and also can be used as a logical unit.

Logical System Board (LSB)

A logical unit name assigned to an XSB. Each domain has its own set of LSB assignments. LSB numbers are used to control how resources such as kernel memory get allocated within domains.

1-4 SPARC Enterprise Mx000 Servers Dynamic Reconfiguration User’s Guide • January 2012

TABLE 1-2 Terms Related to Hardware Configurations (Continued)

Domain A

Domain B

System board #0

System board #1

System board #2

System board #3

Domain B

System board #1

System board #3

System board #0

System board #2

Term Definition

System board The hardware resources of a PSB or an XSB. A system board is used

to describe the hardware resources for operations such as domain construction and identification. In this manual, this refers to the XSB.

Uni-XSB One of the division types of a PSB. Uni-XSB is a name for when a PSB

is logically only one unit (undivided status). It is a default value setting for the division type for a PSB. The division type can be changed by using the XSCF command setupfru(8). Uni-XSB may be used to describe a PSB division type or status.

Quad-XSB One of the division types of a PSB. Quad-XSB is a name for when a

PSB is logically divided into four parts. The division type can be changed by using the XSCF command setupfru(8). Quad-XSB may be used to describe a PSB division type or status.

1.2 Basic DR Functions

This section describes the basic DR functions.

FIGURE 1-3 shows DR processing.

FIGURE 1-3 DR Processing Flow

Chapter 1 Overview of Dynamic Reconfiguration 1-5

In the example shown in FIGURE 1-3, system board #2 is deleted from domain A and added to domain B. In this way, the physical configuration of the hardware (mounting locations) is not changed but the logical configuration is changed for management of the system boards.

1.2.1 Adding a System Board

You can use DR to add a system board to a domain provided that board is installed in the system and not assigned to another domain. You can do so without stopping the Oracle Solaris OS running in the domain.

A system board is added in such stages as connect, and configure.

In the add operation, the selected system board is connected to the target domain. Then, the system board is configured to the Oracle Solaris OS of the domain. At this point, addition of the system board is completed.

1.2.2 Deleting a System Board

You can use DR to delete a system board from a domain without stopping the Oracle Solaris OS running in that domain.

A system board is deleted in such stages as unconfigure and disconnect. If the board must be assigned to another domain, the delete operation must also include an unassign step.

In the delete operation, the selected system board is unconfigured from its domain by the Oracle Solaris OS. Then, the board is disconnected from the domain. At this point, deletion of the system board is completed.

1.2.3 Moving a System Board

You can use DR to reassign a system board from one domain to another without stopping the Oracle Solaris OS running in either domain.

This move function can change the configurations of both domains without physical removal and remounting of the system board.

The move operation for a system board is a serial combination of the “delete” and “add” operations. In other words, the selected system board is deleted from its domain and then added to the target domain.

1-6 SPARC Enterprise Mx000 Servers Dynamic Reconfiguration User’s Guide • January 2012

1.2.4 Replacing a System Board

You can use DR to remove a system board from a domain and either add it back later, or replace it with another system board, provided both boards satisfy DR requirements as described in this document. You can do so without stopping the Oracle Solaris OS running in either domain.

You can replace system board in the case of exchanging hardware resources such as CPUs, memory, I/O devices.

A system board is replaced successively in stages.

In the replace operation, the selected system board is deleted from the OS of the domain. Then, the system board is removed when it is ready to be released from its domain. After field parts replacement or other such task, the system board is re-installed and added.

Note – You cannot use DR to replace a system board in a midrange server because

doing so would replace an MBU. To replace a system board in a midrange server, you must turn off the power of all domains, then replace the board without using DR commands.

1.3 Security

DR operations are executed based on privileges. For information about privileges and user accounts, see the SPARC Enterprise M3000/M4000/M5000/M8000/M9000 Servers Administration Guide.

1.4 Overview of DR User Interfaces

DR operations are performed through the command line interface (CLI) within the XSCF shell or through the browser-based user interface (BUI) in the XSCF Web provided by the eXtended System Control Facility (XSCF). These operations are collectively managed by the XSCF. Furthermore, XSCF security management restricts DR operations to administrators who have the proper access privileges.

Chapter 1 Overview of Dynamic Reconfiguration 1-7

For details of XSCF shell commands provided for DR, see Section 3.1, “How To Use

the DR User Interface” on page 3-1. XSCF Web is beyond the scope of this document.

See the SPARC Enterprise M3000/M4000/M5000/M8000/M9000 Servers XSCF User’s Guide for further information.

1-8 SPARC Enterprise Mx000 Servers Dynamic Reconfiguration User’s Guide • January 2012

CHAPTER

What You Must Know Before Using DR

This chapter provides information you must know to successfully use the DR functions.

This chapter includes these sections:

■ Section 2.1, “System Configuration” on page 2-1

■ Section 2.2, “Conditions and Settings Using XSCF” on page 2-13

■ Section 2.3, “Conditions and Settings Using Oracle Solaris OS” on page 2-16

■ Section 2.4, “Status Management” on page 2-18

■ Section 2.5, “Operation Management” on page 2-27

2.1 System Configuration

This section describes the conditions, premises, and actions for operating the DR functions to construct a system.

2.1.1 System Board Components

There are three types of system board components that can be added and deleted by DR: CPU, memory, and I/O device. system board of a midrange server that is divided into one Uni-XSB, and into Quad-XSBs. high-end server that is divided into one Uni-XSB, and into Quad-XSBs.

FIGURE 2-3 and FIGURE 2-4 show examples of a system board of a

FIGURE 2-1 and FIGURE 2-2 show examples of a

2-1

Note – Due to diagnostic requirements, the DR function works only on boards that

CMU IOU

Memory

XSB 00-0

I/O device

Memory

XSB 01-0

I/O device

MBU

have at least one CPU and memory.

FIGURE 2-1 Example of Hardware Configuration (with Uni-XSB of Midrange Server)

2-2 SPARC Enterprise Mx000 Servers Dynamic Reconfiguration User’s Guide • January 2012

FIGURE 2-2 Example of Hardware Configuration (with Quad-XSBs of Midrange Server)

CMU IOU

Memory

XSB 00-2

Memory

XSB 00-3

XSB 00-0

Memory

I/O device

XSB 00-1

Memory

I/O device

Memory

XSB 01-2

Memory

XSB 01-3

XSB 01-0

Memory

I/O device

XSB 01-1

Memory

I/O device

MBU

Chapter 2 What You Must Know Before Using DR 2-3

FIGURE 2-3 Example of a Hardware Configuration (with Uni-XSBs of High-end Server)

CMU IOU

Memory

XSB 00-0

I/O device

CMU IOU

XSB 00-2

XSB 00-3

XSB 00-0

Memory

I/O device

XSB 00-1

Memory

I/O device

Memory

I/O device

Memory

I/O device

FIGURE 2-4 Example of a Hardware Configuration (with Quad-XSBs of High-end Server)

2.1.1.1 CPU

Using DR to change a CPU configuration is easier than using it to change the configuration of memory or an I/O device. An added CPU is automatically recognized by the Oracle Solaris OS and becomes

2-4 SPARC Enterprise Mx000 Servers Dynamic Reconfiguration User’s Guide • January 2012

available for use.

A CPU to be deleted must meet the following conditions:

■ No running process is bound to the CPU to be deleted. If a running process is

bound to the target CPU, you must unbind or stop the process.

■ The CPU to be deleted does not belong to any processor set. If the target

processor belongs to a processor set, you must delete the CPU from the processor set by using the psrset(1M) command.

■ If the resource pools facility is in use by the domain, the CPU cannot be deleted

unless the minimum processor set sizes can otherwise be maintained. Use the Oracle Solaris commands pooladm(1M) and poolcfg(1M) to check these parameters and, if necessary, adjust the sizes of the domain's resource pools.

Note – These conditions also apply to movement of a system board.

If any of the above conditions are not met, the DR operation is stopped and a message is displayed. However, if you specify the deleteboard(8) command with the -f (force) option, these protections are ignored and DR continues the deletion process.

Note – Exercise care when using the -f (force) option, as doing so introduces risk of

domain failure.

To avoid this problem and automate the operations for CPUs, the Oracle Solaris OS provides the Reconfiguration and Coordination Manager (RCM) script function. For details of RCM, see Section 3.4, “RCM Script” on page 3-27.

For information about mixed configurations of SPARC64 VII+, SPARC64 VII, and SPARC64 VI processors, see Section 2.5.9, “SPARC64 VII+, SPARC64 VII, and

SPARC64 VI Processors and CPU Operational Modes” on page 2-30.

2.1.1.2 Memory

The DR functions classify system boards by memory usage into two types:

■ Kernel memory board

■ User memory board

Chapter 2 What You Must Know Before Using DR 2-5

(1) Kernel Memory Board

A kernel memory board is a system board on which kernel memory (memory internally used by the Oracle Solaris OS and containing an OpenBoot PROM program) is loaded. Kernel memory cannot be removed from the system. But the location of kernel memory can be controlled, and kernel memory can be copied from one board to another.

■ To control whether a system board contains kernel memory, use one or more of

the following features, which are described below: kernel cage, floating boards, and kernel memory assignment.

■ To copy kernel memory from one board to another, use the Copy-rename

operation. Copy-rename makes it possible for you to perform DR operations on kernel memory boards.

(1.1) Kernel Cage

The kernel cage function must be in use for DR operations on memory to succeed. Without the kernel cage, kernel memory could be assigned to all system boards, making it impossible to perform DR operations on memory. With the kernel cage, kernel memory is limited to a minimum set of system boards.

For details on enabling this function, see Section 2.3.2, “Settings of Kernel Cage

Memory” on page 2-17.

(1.2) Floating Boards

A floating board is a system board that is designated to be moved easily to another domain. In general, kernel memory is not assigned to a floating board unless absolutely necessary.

However, kernel memory can be assigned to a floating board when one of the following is true:

■ The total amount of space available among non-floating boards is not enough to

hold the kernel memory.

■ The deleteboard(8) command is used with its -f (force) option.

For details on enabling the floating board option for a system board, see

Section 2.2.2.2, “Floating Board Option” on page 2-14. For further details, also see the

SPARC Enterprise M3000/M4000/M5000/M8000/M9000 Servers XSCF User’s Guide or the setdcl(8) man page.

2-6 SPARC Enterprise Mx000 Servers Dynamic Reconfiguration User’s Guide • January 2012

(1.3) Kernel Memory Assignment

When a domain is powered on, the Power On Self Test (POST) initially assigns an address space to each system board in that domain. The order in which address spaces are assigned depends on the LSB number and floating board option of each system board. The first address spaces are assigned to non-floating boards in ascending order of LSB number. Then, additional address spaces are assigned to floating boards, again in ascending order of their LSB numbers.

When the kernel cage is enabled, kernel memory is assigned to system boards in the order of their address spaces. The kernel cage begins in the first address space (which initially corresponds to the non-floating board with the lowest LSB number). If the kernel requires more memory, then the kernel cage expands to the next address space (which initially corresponds to the non-floating board with the next-lowest LSB number), and so on. The kernel cage extends into the address spaces of floating boards only if kernel memory is too large to fit in the address spaces of the non-floating boards.

Note – During a copy-rename operation, the address spaces initially assigned by

POST are exchanged between system boards. The effects of this process persist through reboots of a domain. Therefore, kernel memory may be assigned in a seemingly different order until the domain has gone through a full poweroff(8) and poweron(8) cycle, as this pair of operations cancels the effects of copy-rename operations.

For details on assigning LSB numbers to system boards, see the SPARC Enterprise M3000/M4000/M5000/M8000/M9000 Servers XSCF User’s Guide or the setdcl(8) man page.

(1.4) Copy-rename

Kernel memory itself cannot be removed, but it can be transferred to another system board. A DR operation to delete a kernel memory board must first perform this transfer, which is called a copy-rename operation.

The Oracle Solaris OS selects the target for the copy-rename operation from among the available user memory boards. The following selection and preference criteria are in effect:

■ The copy-destination board must not yet contain any kernel memory. (It must be

a user memory board.)

■ The copy-destination board must not be a floating board, unless the -f (force)

option is used with the deleteboard(8) command.

■ The copy-destination board must contain at least as much physical memory as the

system board being deleted.

Chapter 2 What You Must Know Before Using DR 2-7

■ If more than one system board satisfies all the selection criteria to the same degree

of satisfaction, the one with the lowest LSB number is selected as the copy-destination board.

Note – If no system boards meet the selection criteria, the DR operation to delete the

kernel memory board will fail.

Once the copy-destination board has been selected, the Oracle Solaris OS performs a memory deletion on the selected user memory board.

Then, the kernel memory on the system board to be deleted is copied into memory on the selected copy-destination system board. The system is suspended while the copying is in progress. After all the memory is copied, the address space of the copy-destination board is renamed to that of the kernel memory board being deleted.

Note – If the address space of a system board is renamed by a copy-rename

operation, the change will persist across reboots of the domain. A poweroff(8)/poweron(8) cycle of the domain will reset the address space assignments and remove the effects of one or more copy-rename operations.

(2) User Memory Board

A user memory board is a system board on which no kernel memory is loaded. Before deleting user memory, the system attempts to swap out the physical pages to the swap area. Sufficient swap space must be available for this operation to succeed.

(2.1) Locked Pages and ISM Pages

Some user pages are locked into memory and cannot be swapped out. These pages receive special treatment by DR.

Intimate Shared Memory (ISM) pages are special user pages which are shared by all processes. ISM pages are permanently locked and cannot be swapped out as memory pages. ISM is usually used by Data Base Management System (DBMS) software to achieve better performance.

Although locked pages cannot be swapped out, the system automatically moves them to the memory on another system board to avoid any problem concerning the pages. Note, however, that the deletion of user memory fails if there is not sufficient free memory size on the remaining system boards to hold the relocated pages.

2-8 SPARC Enterprise Mx000 Servers Dynamic Reconfiguration User’s Guide • January 2012

+ 118 hidden pages

Fujitsu SPARC Enterprise M4000, SPARC Enterprise M5000, SPARC Enterprise M8000, SPARC Enterprise M9000 Features Manual

Specifications and Main Features

Frequently Asked Questions

User Manual

Contents

Preface

Audience

Related Documentation

Text Conventions

Syntax of the Command-Line Interface (CLI)

Documentation Feedback

Overview of Dynamic Reconfiguration

1.1 DR

1.2 Basic DR Functions

1.2.1 Adding a System Board

1.2.2 Deleting a System Board

1.2.3 Moving a System Board

1.2.4 Replacing a System Board

1.3 Security

1.4 Overview of DR User Interfaces

What You Must Know Before Using DR

2.1 System Configuration

2.1.1 System Board Components

2.1.1.1 CPU

2.1.1.2 Memory