IBM 88553RX, 88554RU, Eserver xSeries 455 Installation Manual

ibm.com/redbooks

IBM Eserver xSeries 455 Planning and Planning and Installation Guideuide

David Watts

Aubrey Applewhaite

Yonni Meza

Describes the technical details of the new 64-bit server

Covers supported Windows and Linux 64-bit operating systems

Helps you prepare for and perform an installation

Front cover

IBM Eserver xSeries 455 Planning and Installation Guide

February 2004

International Technical Support Organization

SG24-7056-00

Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

First Edition (February 2004)

This edition applies to the IBM Eserver xSeries 455, machine type 8855.

Note: Before using this information and the product it supports, read the information in “Notices” on page vii.

Contents

Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii

Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .ix

The team that wrote this redbook. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

Become a published author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi

Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii

Chapter 1. Technical description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.1 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.1.1 Comparing the x455 with the x450 . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.1.2 Features not supported . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.2 The x455 base models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.2.1 Front and rear views . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.3 System assembly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

1.4 IBM XA-64 chipset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.4.1 The processor-board assembly. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

1.4.2 The memory-board assembly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

1.4.3 PCI-X board assembly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

1.5 Remote Supervisor Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

1.6 RXE-100 Expansion Enclosure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

1.7 Multinode scalable partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

1.7.1 RXE-100 connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

1.7.2 Multinode configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

1.7.3 Integrated I/O function support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

1.7.4 Error recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

1.8 Redundancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

1.9 Light path diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

1.10 Extensible Firmware Interface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

1.10.1 GUID Partition Table disk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

1.10.2 EFI System Partition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

1.10.3 EFI and the reduced-legacy concept . . . . . . . . . . . . . . . . . . . . . . . 34

1.11 Operating system support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

1.12 Enterprise X-Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

1.12.1 NUMA architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

Chapter 2. Positioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

2.1 Migrating to a 64-bit platform. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

2.2 Scalable system partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

iv IBM Eserver xSeries 455 Planning and Installation Guide

2.2.1 RXE-100 Expansion Enclosure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

2.3 Operating system support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

2.4 Server consolidation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

2.5 ServerProven® . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

2.6 IBM Datacenter Solution Program. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

2.7 Application solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

2.7.1 Database applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

2.7.2 Business logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

2.7.3 e-Business and security transactions . . . . . . . . . . . . . . . . . . . . . . . . 50

2.7.4 In-house developed compute-intensive applications . . . . . . . . . . . . 51

2.7.5 Science and technology industries . . . . . . . . . . . . . . . . . . . . . . . . . . 51

2.8 Why choose x455 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

Chapter 3. Planning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

3.1 System hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

3.1.1 Processors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

3.1.2 Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

3.1.3 PCI-X slot configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

3.1.4 Broadcom Gigabit Ethernet controller . . . . . . . . . . . . . . . . . . . . . . . . 64

3.2 Cabling and connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

3.2.1 SMP Expansion connectivity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

3.2.2 Remote Expansion Enclosure connectivity . . . . . . . . . . . . . . . . . . . . 70

3.2.3 Remote Supervisor Adapter connectivity . . . . . . . . . . . . . . . . . . . . . 76

3.2.4 Serial connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

3.3 Storage considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

3.3.1 xSeries storage solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

3.3.2 Tape backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

3.4 Rack installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

3.5 Power considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

3.6 Operating system support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

3.6.1 Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

3.7 IBM Director support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

3.8 Solution Assurance Review. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

3.8.1 Trigger Tool. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

3.8.2 Electronic Solution Assurance Review (eSAR). . . . . . . . . . . . . . . . . 90

Chapter 4. Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

4.1 Using The Extensible Firmware Interface . . . . . . . . . . . . . . . . . . . . . . . . . 92

4.1.1 EFI Firmware Boot Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

4.1.2 The EFI shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

4.1.3 Driver Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

4.1.4 Flash update . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112

4.1.5 Configuration/Setup utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

Contents v

4.1.6 Diagnostic utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

4.1.7 Boot Option Maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122

4.2 Configuring scalable partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

4.2.1 Creating a scalable partition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

4.2.2 Booting a scalable partition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134

4.2.3 Multiple Monitors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

4.2.4 Deleting a scalable partition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

4.3 Installing Windows Server 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

4.3.1 Important information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

4.3.2 Preparing to install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

4.3.3 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

4.3.4 Post-setup phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

4.4 Installing Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

4.4.1 Linux IA-64 kernel overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152

4.4.2 Choosing a Linux distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153

4.4.3 Installing SUSE LINUX Enterprise Server 8.0. . . . . . . . . . . . . . . . . 156

4.4.4 Installing Red Hat Enterprise Linux AS . . . . . . . . . . . . . . . . . . . . . . 160

4.4.5 Linux boot process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162

4.4.6 Information about the installed system . . . . . . . . . . . . . . . . . . . . . . 163

4.4.7 Using the serial port for the Linux console . . . . . . . . . . . . . . . . . . . 171

4.4.8 RXE-100 Expansion Enclosure. . . . . . . . . . . . . . . . . . . . . . . . . . . . 172

4.4.9 Upgrading drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173

Chapter 5. Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175

5.1 IBM Director . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176

5.1.1 Scalable Systems Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177

5.2 The Remote Supervisor Adapter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177

5.2.1 Connecting via a Web browser . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179

5.2.2 Connecting via the ASM interconnect . . . . . . . . . . . . . . . . . . . . . . . 184

5.2.3 Installing the device driver. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185

5.2.4 Configuring the remote control password . . . . . . . . . . . . . . . . . . . . 186

5.3 Management using the Remote Supervisor Adapter . . . . . . . . . . . . . . . 187

5.3.1 Configuring which alerts to monitor . . . . . . . . . . . . . . . . . . . . . . . . . 188

5.3.2 Configuring SNMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190

5.3.3 Sending alerts directly to IBM Director . . . . . . . . . . . . . . . . . . . . . . 191

5.3.4 Creating a test event action plan in IBM Director . . . . . . . . . . . . . . 193

5.4 Windows System Resource Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . 196

5.4.1 WSRM description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197

5.4.2 WSRM features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198

5.4.3 WSRM in the x455 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199

Abbreviations and acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201

Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203

vi IBM Eserver xSeries 455 Planning and Installation Guide

IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203

Other publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203

Online resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204

How to get IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208

Help from IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209

Notices

This information was developed for products and services offered in the U.S.A.

IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to:

IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A.

The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES

THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.

IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you.

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.

This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental.

COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrates programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and distribute these sample programs in any form without payment to IBM for the purposes of developing, using, marketing, or distributing application programs conforming to IBM's application programming interfaces.

viii IBM Eserver xSeries 455 Planning and Installation Guide

Trademarks

The following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both:

Chipkill™ DB2 Connect™ DB2 Universal Database™ DB2® DRDA® Enterprise Storage Server® ESCON®

Eserver™

Eserver™ eServer™ FlashCopy® FICON™

IBM® ibm.com® iSeries™ LANClient Control Manager™ Notes® OnForever™ Predictive Failure Analysis® PS/2® pSeries® Redbooks™ Redbooks (logo) ™ RETAIN®

ServerGuide™ ServerProven® ServeRAID™ ThinkPad® Tivoli® TotalStorage® Wake on LAN® X-Architecture™ xSeries® zSeries®

The following terms are trademarks of International Business Machines Corporation and Rational Software Corporation, in the United States, other countries or both.

Rational®

The following terms are trademarks of other companies:

Intel, Intel Inside (logos), MMX, and Pentium are trademarks of Intel Corporation in the United States, other countries, or both.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

SET, SET Secure Electronic Transaction, and the SET Logo are trademarks owned by SET Secure Electronic Transaction LLC.

Other company, product, and service names may be trademarks or service marks of others.

Preface

The IBM Eserver xSeries® 455 is the second generation Enterprise X-Architecture™ server using the 64-bit IBM® XA-64 chipset and the Intel® Itanium 2 processor. Unlike the x450, its predecessor, the x455 supports the merging of four server chassis to form a single 16-way image, providing even greater expandability and investment protection.

This IBM Redbook is a comprehensive resource on the technical aspects of the server, and is divided into five key subject areas:

 Chapter 1, “Technical description” on page 1, introduces the server and its

subsystems and describes the key features and how they work. This includes the Extensible Firmware Interface, which provides a powerful replacement to the BIOS facility found on the IA-32 platform.

 Chapter 2, “Positioning” on page 39, examines the types of applications that

would be used on a server such as the x455.

 Chapter 3, “Planning” on page 55, describes the considerations when

planning to purchase and planning to install the x455. It covers such topics as configuration, operating system specifics, scalability, and physical site planning.

 Chapter 4, “Installation” on page 91, covers the process of installing

Windows® Server 2003, SUSE LINUX Enterprise Server, and Red Hat Enterprise Linux AS on the x455.

 Chapter 5, “Management” on page 175, describes how to use the Remote

Supervisor Adapter to send alerts to an IBM Director management environment.

The team that wrote this redbook

This redbook was produced by a team of specialists from around the world working at the International Technical Support Organization, Raleigh Center.

David Watts is a Consulting IT Specialist at the International Technical Support Organization in Raleigh. He manages residencies and produces IBM Redbooks™ on hardware and software topics related to IBM xSeries systems and associated client platforms. He has authored more than 30 IBM Redbooks and redpapers; his most recent books include Implementing Systems Management Solutions Using IBM Director, SG24-6188. He has a

x IBM Eserver xSeries 455 Planning and Installation Guide

Bachelor of Engineering degree from the University of Queensland (Australia) and has worked for IBM for more than 14 years. He is an IBM Eserver™ Certified Specialist for xSeries and an IBM Certified IT Specialist.

Aubrey Applewhaite is a Senior IT Specialist working for the IBM Systems Group in the United Kingdom. He is a member of the Server Implementation Team and specializes in xSeries hardware, Microsoft® Windows, clustering and VMware. He has worked in the IT industry for over 16 years and been at IBM for eight years. He currently works in a customer-acing role providing consultancy and practical assistance to help IBM customers implement new technology with particular emphasis on xSeries hardware. He holds a Bachelor of Science Degree in Sociology and Politics from Aston University and is an MCSE for both Windows NT® and Windows 2000, IBM eServer™ Certified Systems Expert, Cisco CNA, and Compaq Proliant ASE.

Yonni Meza is an xSeries Specialist in Peru who also works as the country’s PCI Instructor, teaching courses on xSeries and Personal Computer Division (PCD) products. Additionally, he supports the PCD team with pre- and post-sales technical support, conducting demos and presentations on a regular basis. He has four years of experience in personal computing systems as well as Intel servers. Furthermore, he has implemented several xSeries solutions such as clustering in Windows and Linux with SCSI, SAN and ESS. Yonni studied Systems Engineering at the University of Lima.

The redbook team (l-r): David, Aubrey, Yonni

Preface xi

Thanks to the following people for their contributions to this project:

Henry Artner, Service Education Curriculum Manager, Raleigh Pat Byers, Program Director, Linux xSeries Alliances & Marketing Alex Candelaria, IBM Center for Microsoft Technologies, Seattle Greg Clarke, IBM Advanced Technical Support, Dallas Rufus Credle, International Technical Support Organization, Raleigh Gary Hade, IBM Linux Technology Center, Beaverton Jim Hanna, xSeries development, Austin Cecil Lockett, Senior Engineer, Engineering Software, Raleigh Gerry McGettigan, Advanced Technical Support, EMEA Michael L Nelson, IBM Eserver Solutions Engineering, Raleigh Lubos Nikolini, Systems Engineer, HT Computers Charles Perkins, Course Developer, Service and Support Education, Raleigh Steve Powell, Service and Support Education Team, Raleigh Ken Rauch, Delivery Project Manager, Markham Jose Rodriquez Ruibal, Advanced Technical Support, EMEA Steve Russell, EMEA ATS xSeries Product Introduction Center, Hursley Bob Zuber, x455 World Wide Product Manager, Raleigh Julie Czubik, Technical Editor, ITSO, Poughkeepsie

Become a published author

Join us for a two- to six-week residency program! Help write an IBM Redbook dealing with specific products or solutions, while getting hands-on experience with leading-edge technologies. You'll team with IBM technical professionals, Business Partners and/or customers.

Your efforts will help increase product acceptance and customer satisfaction. As a bonus, you'll develop a network of contacts in IBM development labs, and increase your productivity and marketability.

Find out more about the residency program, browse the residency index, and apply online at:

ibm.com/redbooks/residencies.html

xii IBM Eserver xSeries 455 Planning and Installation Guide

Comments welcome

Your comments are important to us!

We want our Redbooks to be as helpful as possible. Send us your comments about this or other Redbooks in one of the following ways:

 Use the online Contact us review redbook form found at:

ibm.com/redbooks

 Send your comments in an Internet note to:

redbook@us.ibm.com

 Mail your comments to:

IBM Corporation, International Technical Support Organization Dept. HZ8 Building 662 P.O. Box 12195 Research Triangle Park, NC 27709-2195

Chapter 1. Technical description

The IBM ^ xSeries 455 is the latest IBM top-of-the-line server and is the second implementation of the 64-bit IBM XA-64 chipset, code named “Summit”, which forms part of the Enterprise X-Architecture strategy. The x455 completes the xSeries product family, leveraging the proven Enterprise X-Architecture to deliver robust and reliable 64-bit systems.

 “Features” on page 2  “The x455 base models” on page 4  “System assembly” on page 6  “IBM XA-64 chipset” on page 7  “Remote Supervisor Adapter” on page 19  “RXE-100 Expansion Enclosure” on page 21  “Multinode scalable partitions” on page 22  “Redundancy” on page 27  “Light path diagnostics” on page 27  “Extensible Firmware Interface” on page 29  “Operating system support” on page 35  “Enterprise X-Architecture” on page 35

2 IBM Eserver xSeries 455 Planning and Installation Guide

1.1 Features

The following are the key features of the x455:  One-way or two-way Intel Itanium 2 models, upgradable to 4-way in a single

node and 16-way in a 4-node partition.

 64 MB XceL4 Server Accelerator Cache providing an extra level of cache,

upgradeable to 256 MB in a 4-node partition.

 1 GB or 2 GB RAM standard, upgradeable to 56 GB in a single node and 224

GB in a 4-node partition. Available options are 512 MB, 1 GB, and 2 GB ECC DDR SDRAM RDIMMs.

 Memory enhancement such as memory mirroring, Chipkill™, Memory

ProteXion, and hot swap.

 Dual channel Ultra320 SCSI/RAID controller.  Six 64-bit Active PCI-X slots: Two 133 MHz, two 100 MHz, and two 66 MHz,

upgradeable to 24 x 64-bit PCI-X slots in a four-node partition.

 Scalable system partitioning in two-node and four-node configurations via

three scalability ports.

 Connectivity to an RXE-100 external enclosure for an additional 12 PCI-X

slots, upgradeable to 24 additional PCI-X slots in a 4-node partition.

 Two hot-swap 1-inch drive bays, upgradeable to eight in a four-node partition.  Support for major storage subsystems, including SCSI and Fibre Channel.  Light path diagnostics for troubleshooting.  Remote Supervisor Adapter (RSA) for systems management and remote

diagnostics.

 Integrated dual 10/100/1000 Mbps Ethernet controller.  Integrated ATI Rage XL with 8 MB video RAM.  Three USB ports and one serial port.  Two 1050 W hot swap power supply.  24x combination DVD/CD-RW drive.  4U x 26” rack drawer design.

1.1.1 Comparing the x455 with the x450

The x455 builds on the proven and popular x450 and brings a number of enhancements. Table 1-1 on page 3 summarizes the differences and enhancements between the two servers.

Chapter 1. Technical description 3

Table 1-1 Comparing the differences between the x455 and x450

1.1.2 Features not supported

Due to its 64-bit architecture many existing 32-bit applications are no longer supported. These include:

 32-bit and 16-bit operating systems  ServerGuide™  Remote Deployment Manager (RDM)  LANClient Control Manager™ (LCCM)  UpdateExpress  Access Support

64-bit versions of some of these tool will be made available in the future.

The following functions are also not supported:

 More than one RXE-100 connected to a single node  Hot add/remove of an RXE-100  Physical partitioning within a single node  Partial mirroring of memory  Hot add/swap of the CD-ROM or diskette drive (if installed)  Inter-Process Communications (IPC) over scalability ports  Hot adding memory (hot swap is supported)  PS/2® keyboard and mouse  Parallel port

Component x450 x455

Maximum memory (GB) 40 56

Active memory with hot-swap support No Yes

Multi chassis support No Yes (one, two or four

chassis)

Shared RXE-100 between 2 machines No Yes

Redundant cabling to RXE-100 (only from a single independent machine)

No Yes

Enterprise X-Architecture First generation

chipset

Second generation chipset

Important: The x455 does not have PS/2 ports for a keyboard and mouse. Either a USB keyboard and mouse are required or the appropriate cables to connect to a KVM switch.

4 IBM Eserver xSeries 455 Planning and Installation Guide

1.2 The x455 base models

Powered by XA-64 Enterprise X-Architecture and the 64-bit Itanium 2 “Madison” processors, the x455 server brings the future of 64-bit processing and production-level reliability to your data centers today. Featuring mainframe-inspired advanced mission-critical functions, you can depend on these 16-way-capable enterprise servers to run your complex business applications around the clock.

The initial models of the x455 are listed in Table 1-2.

Table 1-2 Initial x455 base models

The base models can also be connected together to form two-node (eight CPUs) and four-node (16 CPUs) configurations. See “Multinode scalable partitions” on page 22 for details.

1.2.1 Front and rear views

Figure 1-1 on page 5 shows the front view of the x455 showing the system components.

Base model 8855-1RX 8855-2RX 8855-3RX

Itanium 2 processors 1 x 1.3 GHz 2 x 1.4 GHz 2 x 1.5 GHz

Max SMP 4-way 4-way 4-way

Memory 1 GB 2 GB 2 GB

L1 cache 32 KB 32 KB 32 KB

L2 cache 256 KB 256 KB 256 KB

L3 cache 3 MB 4 MB 6 MB

XceL4 Accelerator Cache 64 MB 64 MB 64 MB

Chapter 1. Technical description 5

Figure 1-1 Front panel of the xSeries 455

Figure 1-2 shows the rear view of the x455 showing the system connectors.

Figure 1-2 Rear view of the x455

Power button

Reset button

Power-on light

Hot-swap fans

USB port

System-error light (amber) Information light (amber) SCSI activity light (green) Locator light (blue)

DVD/CD-RW drive

Hot swap power supplies

Blank media bay

Light Path Diagnostics panel (pulls out)

Hot swap drive bays

System power connector (1)

System power connector (2)

RXE Expansion Port (B) connector

Remote Supervisor Adapter connectors and LEDs

Ethernet LEDs

Gigabit Ethernet connectors

RXE Expansion Port (A) connector

Video connector

USB 2 connector

USB 1 connector

RXE Management Port connector

SCSI connector

Serial connector

SMP Expansion Port 1 connector

SMP Expansion Port 2 connector

SMP Expansion Port 3 connector

6 IBM Eserver xSeries 455 Planning and Installation Guide

1.3 System assembly

The x455 has a similar internal design to the x450. The midplane board (viewed from the front) interfaces with three major assemblies:

 The processor-board assembly

This is located to the right of the midplane board and under the memory-board assembly. It houses the Itanium 2 processors, Cache and Scalability Controller, and the 64 MB of Xcel4 cache.

 The memory-board assembly

This is located to the right of the midplane board and above the processor-board assembly. It houses the memory and memory controller.

 The PCI-X board assembly

This is located to the left of the midplane board. It houses all the PCI-X slots and all other I/O components.

Chapter 1. Technical description 7

Figure 1-3 Memory-board and processor-board assembly locations

1.4 IBM XA-64 chipset

The IBM XA-64 chipset is the product name describing the chipset developed under the code name “Summit” and implemented on the IA-64 platform. A product of the IBM Microelectronics Division, the XA-64 chipset leverages the proven Enterprise X-Architecture chipset used initially in the x440 and applies the same technologies to the IA-64 architecture. The XA-64 chipset comprises the following components:

 Itanium 2 processor(s)  Cache and Scalability Controller

A single controller, code named “Tornado”, located within the processor-board assembly.

N O

T E

F O

R O

P E

I R

F L

E P

L A

F A

I T

I N

T E

F R

N T

B O

N O

T E

F O

R O

P E

I R

F L

E P

A C

E F

N W

T H

I N

2 M

U T

E S

F R

N T

B O

Processor-board assembly

Memory-board assembly

PCI-X slots

8 IBM Eserver xSeries 455 Planning and Installation Guide

 Memory controller

A single memory controller, code named “Cyclone”, located within the memory-board assembly.

 Two PCI-X bridges

Two PCI-X bridges, code named “Winnipeg”, one located on the PCI-X board and the other on the I/O board. These control both the PCI-X and Remote I/O.

Figure 1-4 shows the various IBM XA-64 components in an x455 configuration.

Figure 1-4 xSeries 455 system block diagram

Table 1-3 shows how the bandwidths in Figure 1-4 are calculated.

Table 1-3 Bus speeds

Ultra320

SCSI

Gigabit

Ethernet

Video

3x USB

Serial

RSA

33 MHz66 MHz

64-bit

66 MHz

64-bit

100 MHz

64-bit

133 MHz

Bus A66 MHz

RXE

Expansion

Port A

(1 GBps)

B-100

D-133C-133

IBM XA-64 ("Summit") core chipset

6.4 GBps

64 MB

L4 cache

400 MHz

3.2 GBps

CPU 1 CPU 2

CPU 3 CPU 4

PCI-X bridge PCI-X bridge

RXE Expansion Port B (1 GBps)

1 GBps1 GBps

200 MHz

2-way or 4-way

interleaved DDR

Port 2

3.2 GBps

Port 1

3.2 GBps

DDR

SMI-E

DDR

Memory

controller

SMI-E SMI-E SMI-E

6.4 GBps

Processor-board assembly Memory-board assembly

Cache and

scalability

controller

SMP scalability

ports

From To Bandwidth Calculation

CPUs Cache controller 6.4 GBps 400 MHz x 128-bit data path

L4 Cache Cache controller 6.4 GBps 400 MHz x 128-bit data path

Chapter 1. Technical description 9

1.4.1 The processor-board assembly

The processor-board assembly is located below the memory board. It is held in place by retaining levers, an EMC shield and a retention bracket. For instructions to remove or install please refer to the Option Installation Guide.

Figure 1-5 The processor-board assembly

The power modules shown in Figure 1-5 supply power to the processors and are equivalent to VRMs in other systems.

Processors should be installed in the order 1, 2, 3, 4. The bootstrap processor (BSP) may not necessarily be the processor located in processor socket 1. The Intel Itanium Architecture processors are initialized and tested in parallel. The first processor to complete initialization becomes the BSP.

The CPUs are connected together with a 200 MHz frontside bus, but supply data at an effective rate of 400 MHz using the “dual-pump” design of the Intel Itanium 2 architecture is described in “Intel Itanium 2 processors” on page 10.

SDRAM Memory controller 3.2 GBps 400 MHz x 64-bit data path

Cache controller Memory controller 3.2 GBps 400 MHz x 64-bit data path

From To Bandwidth Calculation

Warning: Be careful when removing or installing the processor-board

assembly or the memory-board assembly. It is possible to damage the midplane if not done correctly.

Processors 1 & 3 (processors 2 & 4 are on the underside of the circuit board)

Power modules for each processor

10 IBM Eserver xSeries 455 Planning and Installation Guide

The processor-board assembly is also equipped with LEDs for light path diagnostics for the following components:

 Each processor  Each power module (“pod”)

In addition, a “remind” button is located on the upper side of the processor-board assembly. Pressing this button while the processor-board assembly is not attached to AC power will illuminate any light path LEDs for 10 seconds that had been lit while the system was under power.

Intel Itanium 2 processors

The Itanium 2 processor used in the x455 is code named “Madison”. It uses a ZIF socket design and its small form factor is what allows up to four processors in a 4U dense machine.

Table 1-4 outlines some of the differences between the Itanium and Itanium 2 processors (both the “Madison” and the earlier “McKinley” processor).

Table 1-4 Itanium v Itanium 2 processors

The Itanium 2 processor has three levels of cache, all of which are on the processor die:

 Level 1 cache, 32 KB

Feature Itanium Itanium 2 “McKinley” Itanium 2 “Madison”

Processor core speed 733 or 800 MHz 900 MHz or 1.0 GHz 1.3, 1.4 or 1.5 GHz

L3 Cache 2 or 4 MB 1.5 or 3 MB 3, 4 or 6 MB

Frontside bus 266 MHz 400 MHz, 128 bit 400 MHz, 128 bit

Frontside bus bandwidth 2.1 GBps 6.4 GBps 6.4 GBps

Pipeline stages 10 8 8

Issue ports 9 11 11

on-board registers 328 328 328

Integer units 3 6 6

Branch units 3 3 3

Floating point units 2 2 2

SIMD units 2 1 1

Load and store units 2 (total) 2 load and 2 store 2 load and 2 store

Chapter 1. Technical description 11

This is new and the “closest” to the processor, and is used to store micro-operations. These are decoded executable machine instructions. It serves these to the processor at rated speed. This additional level of cache saves decode time on cache hits.

 Level 2 cache, 256 KB

This is equivalent to L1 cache on the Pentium® III Xeon.

 Level 3 cache 3–6 MB

This is equivalent to L2 cache on the Pentium III Xeon or the L3 cache on the Pentium Xeon MP processor. Unlike the design of the original Itanium processor, this L3 cache is now on the processor die, greatly improving performance, up to 2 times greater than that of the original Itanium.

The x455 also implements a Level 4 cache as described in “IBM XceL4 Accelerator Cache” on page 12.

Intel has also introduced a number of features associated with its Itanium micro-architecture. These are available in the x455, including:

 400 MHz frontside bus

The Pentium III Xeon processor had a 100 MHz frontside bus that equated to a burst throughput of 800 MBps. With protocols such as TCP/IP, this had been shown to be a bottleneck in high-throughput situations. The Itanium 2 processor improves on this by using a single 200 MHz clock but using both edges of each clock cycle to transmit data. This is shown in Figure 1-6.

Figure 1-6 Dual-pumped frontside bus

This increases the performance of the frontside bus. The end result is an effective burst throughput of 6.4 GBps (128-bit wide data path running at 400 MHz), which can have a substantial impact, especially on TCP/IP-based LAN traffic. This is opposed to the Itanium processor, which had a burst throughput of only 2.1 GBps (64-bit wide data path running at 266 MHz).

 Explicitly Parallel Instruction Computing (EPIC)

EPIC technology, developed by Intel and HP, leads to more efficient, faster processors because it eliminates numerous processing inefficiencies in current processors and attacks the perennial data bottleneck problems by increasing parallelism, rather than simply boosting the raw “clock” speed of the processor.

200 MHz clock

12 IBM Eserver xSeries 455 Planning and Installation Guide

Specifically, in today's 32-bit processors, much of the instruction scheduling (the order in which computing instructions are executed) is done on the chip itself, leading to a great deal of overhead and slowing down overall processor performance. Moreover, today's processors are plagued by instruction flow problems since the processor often has to stop what it is doing and reconstruct the instruction flow due to inherent inefficiencies in instruction handling.

EPIC makes the instruction scheduling more intelligent and handles much of the scheduling off-chip, in the compiler program, before feeding “parallelized” instructions to the Itanium 2 processor for execution. The parallelized instructions allow the chip to process a number of instructions simultaneously, increasing performance.

The Itanium 2 architecture is based on EPIC technology and has the following features:

– Provides faster online transaction processing – Has the capability to execute multiple instructions simultaneously – Enables faster calculations and data analysis – Allows for faster storage and movement of large models (CAD, CAE) – Speeds up simulation and rendering times

For more information about the features of the Itanium 2 processor, go to:

http://www.intel.com/design/itanium2

IBM XceL4 Accelerator Cache

Integrated into the processor-board assembly is 64 MB of Level 4 cache, which is shown in Figure 1-4 on page 8. This XceL4 Server Accelerator Cache provides the necessary extra level of cache to maximize CPU throughput by reducing the need for main memory access under demanding workloads, resulting in an overall enhancement to system performance.

Cache memory is two-way interleaved 200 MHz DDR memory and is faster than the main memory because it is directly connected to the Cache and Scalability Controller and does not have additional latency associated with the large fan-out necessary to support the 28 DIMM slots. Since the data interface to the controller is 400 MHz, peak bandwidth for the XceL4 cache is 6.4 GBps.

The XceL4 Accelerator Cache has been designed with commercial workloads in mind that tend to have high cache hit rates. This effectively boosts performance and compensates in part for the 3.2 GBps bandwidth between the Cache and Scalability Controller and Memory Controller.

Chapter 1. Technical description 13

1.4.2 The memory-board assembly

The x455 memory-board assembly is installed in the top of the server and mounts to the side of the midplane using two retaining levers on the top. This location allows for easy access to all memory DIMMs without having to remove any components from the system.

The memory-board assembly houses 28 DIMM slots. All DIMM slots can be used with 512 MB, 1 GB or 2GB RDIMMs for a maximum of 56 GB. Memory can be hot swapped but not hot added. This function, however, is as much a part of the operating system as the hardware. Check with your operating system Help for details.

Figure 1-7 The memory-board assembly, showing the two memory ports

The memory-board assembly is also equipped with LEDs for light path diagnostics for each DIMM. In addition, the assembly is equipped with LEDs for the following:

 Power to memory port 1  Power to memory port 2  Hot-plug memory enabled

Memory used in the x455 is standard PC2100 ECC DDR SDRAM RDIMMs. The memory is 2-way interleaved; however, 4-way interleaving is also supported when both ports are engaged. Interleaving requires DIMMs to be installed in matched pairs and in specific DIMM sockets (see “Memory” on page 57).

There are 14 DIMM slots in each of the two ports, for a total of 28 DIMMs.

Memory Port 1

Memory Port 2

14 IBM Eserver xSeries 455 Planning and Installation Guide

System memory

DIMMs must be installed in matched pairs, since the DIMMs are two-way interleaved. However, if memory is installed in matched fours (a matched pair in each port), the system automatically detects this and will enable 4-way interleaving. With this, memory access is performed simultaneously from both ports (two separate paths into the memory controller as shown in Figure 1-4 on page 8), leading to improved memory performance.

Figure 1-8 Memory DIMMs are divided into two ports

There are a number of advanced features implemented in the x455 memory subsystem, collectively known as

Active Memory:

 Memory ProteXion

Memory ProteXion, also known as “redundant bit steering”, is the technology behind using redundant bits in a data packet to provide backup in the event of a DIMM failure.

Port 1 Por t 2

Front of server

Chapter 1. Technical description 15

Currently, other industry-standard servers use 8 bits of the 72-bit data packets for ECC functions and the remaining 64 bits for data. However, the x455 (and several other xSeries servers) use an advanced ECC algorithm that is based not on bits but on memory symbols. Symbols are groups of multiple bits, and in the case of the x455, each symbol is 4 bits wide. With two-way interleaved memory, the algorithm needs only three symbols to perform the same ECC functions, thus leaving one symbol free (2 bits on each DIMM).

In the event that a chip failure on the DIMM is detected by memory scrubbing, the memory controller can re-route data around that failed chip through the spare symbol (similar to the hot-spare drive of a RAID array). It can do this automatically without issuing a Predictive Failure Analysis® (PFA) or light path diagnostics alert to the administrator. After the second DIMM failure, PFA and light path diagnostics alerts would occur on that DIMM as normal.

 Memory scrubbing

Memory scrubbing is an automatic daily test of all the system memory that corrects soft errors and reports recoverable errors. An excessive rate of recoverable errors reported triggers Memory ProteXion to replace the failing locations.

 Memory mirroring

Memory mirroring is equivalent to RAID-1 in disk arrays, in that memory is divided in two ports and one port is mirrored to the other (see Figure 1-8 on page 14). If 8 GB is installed, then the operating system sees 4 GB once memory mirroring is enabled (it is disabled by default). All mirroring activities are handled by the hardware without any additional support required from the operating system.

When memory mirroring is enabled the data that is written to memory is stored in two locations. One copy is kept in the port 1 DIMMs, while a second copy is kept in the port 2 DIMMs.

During the execution of the read command, the data is read simultaneously from both ports, and error-free data from either port is forwarded. This provides an extra level of error recovery capability.

When an unrecoverable memory error from one of the memory ports is encountered, good data from the non-failing memory port is forwarded to the system. The failing DIMM is reported and indicated with light path. The failing memory port is then disabled.

Certain restrictions exist with respect to placement and size of memory DIMMs when memory mirroring is enabled. These are discussed in “Memory mirroring” on page 58.

16 IBM Eserver xSeries 455 Planning and Installation Guide

 Chipkill memory

Chipkill is integrated into the XA-64 chipset and does not require special Chipkill DIMMs. Chipkill corrects multiple single-bit errors to keep a DIMM from failing. When combining Chipkill with Memory ProteXion and Active Memory, the x455 provides very high reliability in the memory subsystem. Chipkill memory is approximately 100 times more effective than ECC technology, providing correction for up to 4 bits per DIMM, whether on a single chip or multiple chips.

If a memory chip error does occur, Chipkill is designed to automatically take the inoperative memory chip offline while the server keeps running. The memory controller provides memory protection similar in concept to disk array striping with parity, writing the memory bits across multiple memory chips on the DIMM. The controller is able to reconstruct the “missing” bit from the failed chip and continue working as usual.

Chipkill support is provided in the memory controller and implemented using standard RDIMMs, so it is transparent to the operating system.

In addition, to maintain the highest levels of system availability, if a memory error is detected during POST or memory configuration, the server can automatically disable the failing DIMM and continue operating with reduced memory capacity. You can manually re-enable the memory bank after the problem is corrected via the Setup/Configuration option in the EFI Firmware Boot Manager menu. EFI is

Extensible Firmware Interface, the replacement to BIOS as described in

“Extensible Firmware Interface” on page 29.

Memory ProteXion, memory mirroring, and Chipkill provide multiple levels of redundancy to the memory subsystem. Combining Memory ProteXion with Chipkill enables up to two memory chip failures per memory port (14 DIMMs). Both memory ports could sustain up to four memory chip failures. Memory mirroring provides additional protection with the ability to continue operations with memory module failures.

1. The first failure detected by the Chipkill algorithm on each port does not generate a light path diagnostics error, since Memory ProteXion recovers from the problem automatically.

2. Each memory port could then sustain a second chip failure without shutting down.

3. Provided that memory mirroring is enabled, the third chip failure on that port would send the alert and take the DIMM offline, but keep the system running out of the redundant memory bank.

The combination of these technologies provides the most reliable memory subsystem available.

+ 202 hidden pages

IBM 88553RX, 88554RU, Eserver xSeries 455 Installation Manual

Specifications and Main Features

Frequently Asked Questions

User Manual