Silicon Graphics SGI Altix 450 User Manual

SGI® Altix® 450 System User’s Guide

Document Number 007-4857-002

CONTRIBUTORS

Written by Mark Schwenden Illustrated by Chrystie Danzer Production by Mark Schwenden Engineering contributions by Rich Altmaier, Steve Bowen, Michael T . Brown, Dick Brownell, David Collins, Thomas Crapisi, S teve Dean, Hartmut Gottwald,

Paul Wiley, Gary Meyer, Galen Flunker, Paul Kinyon, Mark Koneazny, William Kellerman, Paul Pedersen, Gregory Thorson, and Gary Spilde.

© 2007 SGI. All rights reserved; provided portions may be cop yright in third parties, as indicated elsewhere herein. No permission is granted to copy , distribute, or create derivative works from the contents of this electronic documentation in any manner, in whole or in part, without the prior written permission of Silicon Graphics, Inc.

LIMITED RIGHTS LEGEND

The software described in this document is "commercial computer software" provided with restricted rights (except as to included open/free source) as specified in the F AR 52.227-19 and/or the DFAR 227.7202, or successive sections. Use beyond license provis ions is a violation of worldwide intellectual property laws, treaties and conventions. This document is provided with limited rights as defined in 52.227-14.

The electronic (software) version of this document was developed at private expense; if acquired under an agreement with the USA government or any contractor thereto, it is acquired as “commercial computer software” subject to the provisions of its applicable license agreement, as specified in (a) 48 CFR

12.212 of the FAR; or, if acquired for Department of Defense units, (b) 48 CFR 227-7202 of the DoD FAR Supplement; or sections succeeding thereto. Contractor/manufacturer is SGI, 1140 East Arques Avenue Sunnyvale, CA 94085.

TRADEMARKS AND ATTRIBUTIONS

SGI, the SGI logo, and Altix are registered trademarks, and NUMAlink and NUMAflex are trademarks, of SGI, in the United States and/or other countries worldwide.

Intel and Itanium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. UNIX is a registered trademark in the United States and other countries, licensed exclusively through X/Open Company, Ltd.

Linux is a registered trademark of Linus Torvalds.

All other trademarks mentioned herein are the property of their respective owners.

Record of Revision

Version Description

-001 August 2006, First release

-002 March 2007, Engineering updates to cover RAID 1, DVD-R/W, 2D graphics and IA2.

007-4857-002 iii

Contents

List of Figures . . . . . . . . . . . . . . . . . . . . . . . . xi

List of Tables . . . . . . . . . . . . . . . . . . . . . . . . xv

About This Guide . . . . . . . . . . . . . . . . . . . . . . .xvii

Audience. . . . . . . . . . . . . . . . . . . . . . . . . .xvii

Important Information . . . . . . . . . . . . . . . . . . . . . .xvii

Chapter Descriptions . . . . . . . . . . . . . . . . . . . . . xviii

Related Publications. . . . . . . . . . . . . . . . . . . . . . . xix

Conventions . . . . . . . . . . . . . . . . . . . . . . . . . xxi

Product Support . . . . . . . . . . . . . . . . . . . . . . . . xxi

Reader Comments . . . . . . . . . . . . . . . . . . . . . . .xxii

1. System Installation . . . . . . . . . . . . . . . . . . . . . . . 1

System Installation Overview . . . . . . . . . . . . . . . . . . . . 2

Safety Precautions . . . . . . . . . . . . . . . . . . . . . . . 3

Hazard Statements . . . . . . . . . . . . . . . . . . . . . . 4

ESD Precautions . . . . . . . . . . . . . . . . . . . . . . 4

Safety Measures . . . . . . . . . . . . . . . . . . . . . . 5

007-4857-002 v

Contents

Installing a Rack . . . . . . . . . . . . . . . . . . . . . . . . 6

Preinstallation Activities . . . . . . . . . . . . . . . . . . . . 6

Site Plan Verification . . . . . . . . . . . . . . . . . . . . 6

Tools Required . . . . . . . . . . . . . . . . . . . . . . 7

Power Receptacle Verification . . . . . . . . . . . . . . . . . . 8

Unloading and Moving System Equipment . . . . . . . . . . . . . . . . 12

Unloading the Equipment from the Truck . . . . . . . . . . . . . . . 12

Inspecting the Shipping Crate . . . . . . . . . . . . . . . . . . 14

Transporting the Shipping Crate . . . . . . . . . . . . . . . . . 15

Removing a Short Rack from the Shipping Crate . . . . . . . . . . . . . . 16

Unpacking and Installing the Tall Rack. . . . . . . . . . . . . . . . . 18

Positioning and Leveling Racks. . . . . . . . . . . . . . . . . . . 21

Unpacking and Inspecting Modules . . . . . . . . . . . . . . . . . . . 22

Installing the IRUs in a Rack. . . . . . . . . . . . . . . . . . . . . 24

Determining Space Requirements . . . . . . . . . . . . . . . . . . 24

Determining Where to Attach Rails in a Rack . . . . . . . . . . . . . . . 25

Rackmounting IRUs with Shelf Rails . . . . . . . . . . . . . . . . . 26

Removing a Rackmounted IRU Module . . . . . . . . . . . . . . . . 28

NUMAlink Cabling the System IRUs . . . . . . . . . . . . . . . . . . 29

Cabling the System IRUs to a Power Source . . . . . . . . . . . . . . . . 31

Connecting an IRU Module to a Power Source (No Rack) . . . . . . . . . . . . 31

Connecting System Modules to a Rack Power Source . . . . . . . . . . . . . 34

Connecting the System Console . . . . . . . . . . . . . . . . . . . . 36

Operating Your System . . . . . . . . . . . . . . . . . . . . . . 38

Powering the System On and Off . . . . . . . . . . . . . . . . . . 38

Preparing to Power On . . . . . . . . . . . . . . . . . . . . 38

Powering On With Optional System Console and L1/L2 Controller . . . . . . . . 39

Powering On Manually or With L1 Console . . . . . . . . . . . . . . 41

Powering Off at the System Console . . . . . . . . . . . . . . . . 42

Powering Off Manually. . . . . . . . . . . . . . . . . . . . 43

L1 Front Panel Controls . . . . . . . . . . . . . . . . . . . . . 44

Operating the L1 Controller . . . . . . . . . . . . . . . . . . . . 45

vi 007-4857-002

Contents

2. Operation Procedures . . . . . . . . . . . . . . . . . . . . . . 47

Precautions . . . . . . . . . . . . . . . . . . . . . . . . . 47

ESD Precaution. . . . . . . . . . . . . . . . . . . . . . . 47

Safety Precautions . . . . . . . . . . . . . . . . . . . . . . 48

System Controller Network . . . . . . . . . . . . . . . . . . . . . 49

Console Connections . . . . . . . . . . . . . . . . . . . . . . 50

Connecting to the System Controller . . . . . . . . . . . . . . . . . . 50

Communicating with the System Controller . . . . . . . . . . . . . . . . 51

Powering the System On and Off . . . . . . . . . . . . . . . . . . . 51

Preparing to Power On . . . . . . . . . . . . . . . . . . . . . 52

Powering On and Off at the Console . . . . . . . . . . . . . . . . . 54

Powering On and Off from the L1. . . . . . . . . . . . . . . . . 54

Powering-On and Off From the L2 . . . . . . . . . . . . . . . . 56

Using Embedded Support Partner (ESP) . . . . . . . . . . . . . . . . . 56

Monitoring Your Server. . . . . . . . . . . . . . . . . . . . . . 57

2D Graphics Video Interface . . . . . . . . . . . . . . . . . . . . 58

Installing Optional Components. . . . . . . . . . . . . . . . . . . . 59

I/O Blade Overview . . . . . . . . . . . . . . . . . . . . . 60

Adding or Removing Disk Drives . . . . . . . . . . . . . . . . . . 61

Reconfigure the System Disk Drive(s) for RAID 1 . . . . . . . . . . . . . 61

3. System Overview . . . . . . . . . . . . . . . . . . . . . . . 67

System Models . . . . . . . . . . . . . . . . . . . . . . . . 68

System Architecture . . . . . . . . . . . . . . . . . . . . . . . 70

System Features . . . . . . . . . . . . . . . . . . . . . . . . 72

Modularity and Scalability . . . . . . . . . . . . . . . . . . . . 72

Distributed Shared Memory (DSM) . . . . . . . . . . . . . . . . . 72

Distributed Shared I/O . . . . . . . . . . . . . . . . . . . . . 73

ccNUMA Architecture . . . . . . . . . . . . . . . . . . . . . 73

Cache Coherency . . . . . . . . . . . . . . . . . . . . . 73

Non-uniform Memory Access (NUMA) . . . . . . . . . . . . . . . 74

Reliability, Availability, and Serviceability (RAS) . . . . . . . . . . . . . 74

007-4857-002 vii

Contents

System Components . . . . . . . . . . . . . . . . . . . . . . . 76

Bay (Unit) Numbering . . . . . . . . . . . . . . . . . . . . . 77

Rack Numbering. . . . . . . . . . . . . . . . . . . . . . . 77

Optional System Components . . . . . . . . . . . . . . . . . . . 78

4. System Control . . . . . . . . . . . . . . . . . . . . . . . . 79

Levels of System Control . . . . . . . . . . . . . . . . . . . . . 80

System Controller Interaction . . . . . . . . . . . . . . . . . . . . 81

L1 Controller . . . . . . . . . . . . . . . . . . . . . . . . . 81

L1 Controller Functions . . . . . . . . . . . . . . . . . . . . . 82

L1 Front Panel Display . . . . . . . . . . . . . . . . . . . . . 82

Ethernet Switch . . . . . . . . . . . . . . . . . . . . . . . . 83

Console Hardware Requirements . . . . . . . . . . . . . . . . . . . 84

Operating the L1 . . . . . . . . . . . . . . . . . . . . . . . . 84

L1 Mode . . . . . . . . . . . . . . . . . . . . . . . . . 85

Viewing System Configuration (from an IRU’s Perspective) . . . . . . . . . . 86

Command Targeting . . . . . . . . . . . . . . . . . . . . 86

Console Mode from L1 . . . . . . . . . . . . . . . . . . . . . 87

L1 Console Selection . . . . . . . . . . . . . . . . . . . . 88

Viewing Information, Warnings, and Error Messages . . . . . . . . . . . . 89

L2 Operation . . . . . . . . . . . . . . . . . . . . . . . . . 89

Configuring an L2’s IP Address. . . . . . . . . . . . . . . . . . . 89

Viewing System Configuration . . . . . . . . . . . . . . . . . . . 91

Setting Command Targeting. . . . . . . . . . . . . . . . . . . . 92

Viewing Information, Warnings, and Error Messages . . . . . . . . . . . . . 94

Powering On, Powering Off, and Resetting the System From the L2 . . . . . . . . . 94

Console Mode from the L2 . . . . . . . . . . . . . . . . . . . . 95

Console Selection . . . . . . . . . . . . . . . . . . . . . . 96

L1 Mode From L2 . . . . . . . . . . . . . . . . . . . . . . 98

Upgrading L1 Firmware . . . . . . . . . . . . . . . . . . . . . . 99

Upgrading L1 Firmware . . . . . . . . . . . . . . . . . . . . 99

5. Rack Information . . . . . . . . . . . . . . . . . . . . . . 101

Overview . . . . . . . . . . . . . . . . . . . . . . . . . 101

viii 007-4857-002

Contents

Altix 42U and 20U Rack Features . . . . . . . . . . . . . . . . . . .102

Technical Specifications. . . . . . . . . . . . . . . . . . . . . .104

6. Maintenance and Upgrade Procedures . . . . . . . . . . . . . . . . .107

Maintenance Precautions and Procedures . . . . . . . . . . . . . . . . .107

Preparing the System for Maintenance or Upgrade . . . . . . . . . . . . .108

Returning the System to Operation. . . . . . . . . . . . . . . . . .108

Installing or Removing Internal Parts . . . . . . . . . . . . . . . . . .1 09

Replacing IRU Components. . . . . . . . . . . . . . . . . . . . .110

Remove and Replace the System Control Display Panel . . . . . . . . . . . .110

Removing or Adding IRU Power Supplies. . . . . . . . . . . . . . . .113

Removing and Replacing IRU Fans (Blowers). . . . . . . . . . . . . . .114

Removing or Replacing a System Blade From the IRU . . . . . . . . . . . .118

Removing and Replacing a Compute/Memory Blade . . . . . . . . . . . . .118

Memory DIMM Group Placement and Guidelines. . . . . . . . . . . . . .121

Remove and Replace the DVD Drive . . . . . . . . . . . . . . . . .123

Overview of PCI Express Operation . . . . . . . . . . . . . . . . . .127

Adding or Replacing PCI/PCI-X Cards . . . . . . . . . . . . . . . . . .128

Installing PCI Cards in the Three-Slot Blade . . . . . . . . . . . . . . .129

Add or Remove a PCI Card in Base IA Blades. . . . . . . . . . . . . . .136

Installing or Replacing a Disk Drive in the IA Blade . . . . . . . . . . . . . .138

Removing a System Disk Drive . . . . . . . . . . . . . . . . . .139

Installing a System Disk Drive . . . . . . . . . . . . . . . . . . .1 40

7. Troubleshooting and Diagnostics . . . . . . . . . . . . . . . . . . .143

Troubleshooting Chart . . . . . . . . . . . . . . . . . . . . . .144

L1 Controller Error Messages . . . . . . . . . . . . . . . . . . . .145

LED Status Indicators . . . . . . . . . . . . . . . . . . . . . .147

IRU Power Supply LEDs . . . . . . . . . . . . . . . . . . . .147

IRU NUMAlink Router Port LEDs. . . . . . . . . . . . . . . . . .147

Compute/Memory Blade LEDs. . . . . . . . . . . . . . . . . . .148

SGI Electronic Support . . . . . . . . . . . . . . . . . . . . . .149

A. Technical Specifications and Pinouts . . . . . . . . . . . . . . . . . .153

System-level Specifications . . . . . . . . . . . . . . . . . . . . .153

007-4857-002 ix

Contents

Physical Specifications . . . . . . . . . . . . . . . . . . . . . 154

Environmental Specifications . . . . . . . . . . . . . . . . . . . 156

Power Specifications . . . . . . . . . . . . . . . . . . . . . . 157

I/O Port Specifications . . . . . . . . . . . . . . . . . . . . . 158

Ethernet Port. . . . . . . . . . . . . . . . . . . . . . . 159

Serial Ports . . . . . . . . . . . . . . . . . . . . . . . 160

External SAS/SATA Port Connector . . . . . . . . . . . . . . . . 162

USB Type A Connector . . . . . . . . . . . . . . . . . . . . 163

B. Safety Information and Regulatory Specifications . . . . . . . . . . . . . 165

Safety Information . . . . . . . . . . . . . . . . . . . . . . 165

Regulatory Specifications . . . . . . . . . . . . . . . . . . . . 167

CMN Number . . . . . . . . . . . . . . . . . . . . . . 167

CE Notice and Manufacturer’s Declaration of Conformity . . . . . . . . . . . 167

Electromagnetic Emissions . . . . . . . . . . . . . . . . . . . 168

FCC Notice (USA Only) . . . . . . . . . . . . . . . . . . 168

Industry Canada Notice (Canada Only) . . . . . . . . . . . . . . 169

VCCI Notice (Japan Only) . . . . . . . . . . . . . . . . . . 169

Chinese Class A Regulatory Notice . . . . . . . . . . . . . . . 169

Korean Class A Regulatory Notice. . . . . . . . . . . . . . . . 169

Shielded Cables . . . . . . . . . . . . . . . . . . . . . . 170

Electrostatic Discharge . . . . . . . . . . . . . . . . . . . . 170

Laser Compliance Statements . . . . . . . . . . . . . . . . . . 171

Lithium Battery Statements . . . . . . . . . . . . . . . . . . . 172

Index . . . . . . . . . . . . . . . . . . . . . . . . . . 173

x 007-4857-002

List of Figures

Figure 1-1 30-amp Single-phase Power Receptacle for North American Sites . . . 9

Figure 1-2 32-amp Single-phase Power Plug for International Sites . . . . . 11

Figure 1-3 Dimensions of Tall Rack Shipping Crate . . . . . . . . . 13

Figure 1-4 Dimensions of Short Rack Shipping Crate . . . . . . . . . 14

Figure 1-5 Removing a Short Rack from the Shipping Crate . . . . . . . 17

Figure 1-6 Removing a Tall Rack from the Shipping Crate . . . . . . . . 19

Figure 1-7 Reattaching the Tall Rack Doors. . . . . . . . . . . . 20

Figure 1-8 Leveling Bolts . . . . . . . . . . . . . . . . 21

Figure 1-9 Seismic Tie-down Attachment Points . . . . . . . . . . 22

Figure 1-10 IRU Connection Plate for Desktop Systems . . . . . . . . . 23

Figure 1-11 Mounting Hole Pattern of Rack Rails . . . . . . . . . . 25

Figure 1-12 Installing the Shelf Rails in the Rack . . . . . . . . . . 26

Figure 1-13 Securing the Module to the Front of the Rack . . . . . . . . 27

Figure 1-14 NUMAlink Cabling an IRU to an Additional IRU . . . . . . . 30

Figure 1-15 Power-Supply Connectors on the IRU Module . . . . . . . . 32

Figure 1-16 Connecting an Unracked IRU System to a Power Source . . . . . 33

Figure 1-17 Connecting Multiple Modules to a PDU. . . . . . . . . . 34

Figure 1-18 Optional Three-Phase PDU Example . . . . . . . . . . 35

Figure 1-19 Location of the L1 Console Port . . . . . . . . . . . . 37

Figure 1-20 PDU Circuit Breakers Example . . . . . . . . . . . . 39

Figure 1-21 Location of the Power Button . . . . . . . . . . . . 41

Figure 1-22 L1 Front Panel Functions . . . . . . . . . . . . . 44

Figure 2-1 IRU Power Supply Cable Location Example . . . . . . . . 52

Figure 2-2 Single-Phase Rack PDU Circuit Breaker Switches . . . . . . . 53

Figure 2-3 IRU L1 Controller Display Location Example . . . . . . . . 57

007-4857-002 xi

List of Figures

Figure 2-4 2D Graphics Card Location in IA2 Blade . . . . . . . . . 59

Figure 3-1 SGI Altix 450 System (Short Rack) . . . . . . . . . . . 68

Figure 3-2 SGI Altix 450 Server System (Tall Rack) . . . . . . . . . 69

Figure 3-3 Blade, IRU and Rack Components . . . . . . . . . . . 70

Figure 3-4 Functional Block Diagram of the Individual Rack Unit (IRU) . . . . 71

Figure 3-5 Altix 450 IRU System Components Example . . . . . . . . 77

Figure 4-1 SGI Altix 450 System Control Network (Example) . . . . . . . 80

Figure 4-2 L1 Front Panel . . . . . . . . . . . . . . . . 82

Figure 4-3 Ethernet Switch System Controller Block Diagram (Example) . . . . 83

Figure 5-1 Tall (42U) Altix Rack and Front Lock . . . . . . . . . 103

Figure 5-2 Short (20U) Altix Rack (Rear View) . . . . . . . . . . 104

Figure 6-1 Removing the L1 Controller Panel . . . . . . . . . . 111

Figure 6-2 Replacing the L1 Controller Panel . . . . . . . . . . 112

Figure 6-3 Removing an IRU Power Supply . . . . . . . . . . . 113

Figure 6-4 Replacing an IRU Power Supply . . . . . . . . . . . 114

Figure 6-5 IRU Fan Assembly (Blowers). . . . . . . . . . . . 115

Figure 6-6 Removing a Fan From the IRU . . . . . . . . . . . 116

Figure 6-7 Replacing an IRU Fan. . . . . . . . . . . . . . 117

Figure 6-8 Removing a Compute/Memory Blade From the IRU. . . . . . 119

Figure 6-9 Replacing a Compute/Memory Blade in the IRU. . . . . . . 120

Figure 6-10 DIMM Group Locations Diagram . . . . . . . . . . 122

Figure 6-11 DIMM Memory Slot Location Example . . . . . . . . . 123

Figure 6-12 DVD Removal from IA Blade . . . . . . . . . . . 125

Figure 6-13 DVD Drive Replacement in the IA Blade . . . . . . . . 126

Figure 6-14 Comparison of PCI/PCI-X Connector with PCI Express Connectors . 127

Figure 6-15 Removing a Card Carrier From the Three-Slot Blade . . . . . 129

Figure 6-16 Extracting the Carrier Metal Filler Plate . . . . . . . . . 130

Figure 6-17 Adjusting the Carrier Guide Bar . . . . . . . . . . . 131

Figure 6-18 Mounting Card in Carrier . . . . . . . . . . . . . 132

Figure 6-19 Mounting Half-height PCI Card into Carrier . . . . . . . . 133

Figure 6-20 Moving Carrier Guide Bar to Secure Half-height PCI Card . . . . 133

Figure 6-21 Installing the Bracket to Secure a Half-height PCI Card . . . . . 134

Figure 6-22 Installing a Card in a Slot . . . . . . . . . . . . . 135

xii 007-4857-002

List of Figures

Figure 6-23 Seating the Card in the Slot . . . . . . . . . . . . .1 35

Figure 6-24 Removing the Card Access Panel From the IA Blade . . . . . .136

Figure 6-25 Replacing the IA Blade PCI Access Door . . . . . . . . .137

Figure 6-26 Location of Disk Drive Bays. . . . . . . . . . . . .138

Figure 6-27 Removing a Disk Drive . . . . . . . . . . . . . .139

Figure 6-28 Replacing a Disk Drive . . . . . . . . . . . . . .141

Figure 7-1 Compute Blade Status LED Locations . . . . . . . . . . 1 49

Figure 7-2 Full Support Sequence . . . . . . . . . . . . . .150

Figure A-1 Ethernet Port . . . . . . . . . . . . . . . . .159

Figure A-2 Serial Port Connector. . . . . . . . . . . . . . .160

Figure A-3 Pin Number Locations for External SAS/SATA Port . . . . . .162

Figure A-4 Pin Number Locations for USB Type A Connector . . . . . . .163

Figure B-1 VCCI Notice (Japan Only) . . . . . . . . . . . . .169

Figure B-2 Chinese Class A Regulatory Notice . . . . . . . . . . .169

Figure B-3 Korean Class A Regulatory Notice . . . . . . . . . . .169

007-4857-002 xiii

List of Tables

Table 1-1 Installation Tools. . . . . . . . . . . . . . . . 7

Table 1-2 IRU Space Requirements. . . . . . . . . . . . . . 24

Table 5-1 Tall Altix Rack Technical Specifications . . . . . . . . .104

Table 5-2 Short Altix Rack Technical Specifications . . . . . . . . .104

Table 6-1 Customer-replaceable Components and Maintenance Procedures . . .109

Table 6-2 SGI PCIe Support Levels . . . . . . . . . . . . .128

Table 7-1 Troubleshooting Chart . . . . . . . . . . . . . .144

Table 7-2 L1 Controller Messages . . . . . . . . . . . . . .145

Table 7-3 Power Supply LED States . . . . . . . . . . . . .147

Table A-1 Altix 450 System Configuration Ranges. . . . . . . . . .153

Table A-2 Altix 450 Tall Rack System Physical Specifications . . . . . .154

Table A-3 Altix 450 Short Rack System Physical Specifications . . . . . .155

Table A-4 Environmental Specifications (42U Rack) . . . . . . . . .156

Table A-5 Environmental Specifications (20U Rack) . . . . . . . . .156

Table A-6 Power Specifications . . . . . . . . . . . . . . .157

Table A-7 Ethernet Pinouts . . . . . . . . . . . . . . . .159

Table A-8 Serial Port Pinout. . . . . . . . . . . . . . . .161

Table A-9 SAS/SATA Pin Assignments . . . . . . . . . . . .162

Table A-10 Pin Assignments for USB Type A Connector . . . . . . . .163

007-4857-002 xv

About This Guide

This guide provides an overview of the installation, architecture, general operation and descriptions of the major components that compose the SGI provides the standard procedures for powering on and powering off the system, basic troubleshooting and maintenance information, and important safety and regulatory specifications.

Audience

This guide is written for owners, installers, system administrators, and users of SGI Altix 450 computer systems. It is written with the assumption that the reader has a good working knowledge of computers and computer systems.

Important Information

W arning: To avoid problems that could void your warranty , your SGI or other approved

system support engineer (SSE) should perform any replacement of parts, or service of your SGI Altix 450 system not covered in the following list of items that you can perform yourself:

• Unpacking, setting up and connecting the system rack(s) and individual rack units IRU(s).

• Using your system console and your L1 controller to enter commands and perform system functions such as powering on and powering off, as described in this guide.

Altix® 450 family of systems. It also

• Adding and replacing system blades and DIMM memory as described in this guide.

• Adding and replacing PCI, PCI-X and PCI express cards, as described in this guide.

• Replacing fan modules and power supplies in the individual rack unit (IRU) enclosures.

• Adding and replacing DVD or disk drives in the base IA/IA2 (I/O) blade.

• Using the On/Off switch and other switches (the reset and non-maskable interrupt [NMI] switches) on the system IA or IA2 blade (base I/O).

007-4857-002 xvii

About This Guide

• Using the ESI/ops panel (operating panel) on optional mass storage bricks. See the user’s

Chapter Descriptions

The following topics are covered in this guide:

• Chapter 1, “System Installation” guides an installer through the steps of unpacking and

• Chapter 2, “Operation Procedures,” provides instructions for operation of the system

• Chapter 3, “System Overview,” provides environmental and technical information needed to

• Chapter 4, “System Control,” describes the function of the L1 and L2 controllers and

• Chapter 5, “Rack Information,” describes the rack sizes and features.

• Chapter 6, “Maintenance and Upgrade Procedures,” provides instructions for installing or

• Chapter 7, “Troubleshooting and Diagnostics,” provides recommended actions if problems

• Appendix A, “Technical Specifications and Pinouts‚" provides physical, environmental, and

• Appendix B, “Safety Information and Regulatory Specifications‚" lists all regulatory

guide for your optional SGI mass storage enclosure.

positioning the system rack(s). Rack mounting guidelines and basic cabling for the IRUs is also covered.

including powering on and powering off your system.

properly set up and configure the Altix 450 series system.

provides instructions for operating the controllers.

removing the customer-replaceable components of your system.

occur on your system.

power specifications for your system. Also included are the pinouts for the non-proprietary connectors.

information related to use of the Altix 450 system in the United States and other countries. It also provides a list of safety instructions to follow when installing, operating, or servicing the product.

xviii 007-4857-002

Related Publications

The following SGI documents are relevant to the Altix 450 series system:

• SGI L1 and L2 Controller Software User’s Guide

• SGI InfiniteStorage 120 Mass Storage Hardware Topics

• Reconfigurable Application-Specific Computing User's Guide

Related Publications

(P/N 007-3938-00x) This guide describes how to use the L1 and L2 controller commands at your system console

to monitor and manage your SGI system.

(P/N 007-4853-00x) This document provides a hardware overview of the SGI non-RAID 12-drive optional mass

storage enclosure used with Altix 450 systems.

(P/N 007-4718-00x) This guide discusses the optional SGI reconfigurable application-specific software

computing (RASC) program that delivers scalable, configurable computing elements for the SGI Altix family of servers and superclusters.

• Man pages (online) Man pages locate and print the titled entries from the online reference manuals.

You can obtain SGI documentation, release notes, or man pages in the following ways:

• See the SGI Technical Publications Library at http://docs.sgi.com. Various formats are available. This library contains the most recent and most comprehensive set of online books, release notes, man pages, and other information.

• The release notes, which contain the latest information about software and documentation in this release, are in a file named README.SGI in the root directory of the SGI ProPack for Linux Documentation CD.

• You can also view man pages by typing man <title> on a command line.

SGI systems include a set of Linux man pages, formatted in the standard UNIX “man page” style. Important system configuration files and commands are documented on man pages. These are found online on the internal system disk (or on a DVD) and are displayed using the man command. For example, to display the man page for the xscsidisktest command, type the following on a command line:

man xscsidisktest

007-4857-002 xix

About This Guide

References in the documentation to these pages include the name of the command and the section number in which the command is found.

For additional information about displaying man pages using the man command, see man(1).

In addition, the apropos command locates man pages based on keywords. For example, to display a list of man pages that describe disks, type the following on a command line:

apropos disk

For information about setting up and using apropos, see apropos(1).

xx 007-4857-002

Conventions

The following conventions are used throughout this document:

Convention Meaning

Command This fixed-space font denotes literal items such as commands, files,

routines, path names, signals, messages, and programm in g language structures.

variable The italic typeface denotes variable entries and words or concepts being

defined. Italic typeface is also used for book titles.

user input This bold fixed-space font denotes literal items that the user enters in

interactive sessions. Output is shown in nonbold, fixed-space font. [ ] Brackets enclose optional portions of a command or directive line. ... Ellipses indicate that a preceding element can be repeated. man page(x) Man page section identifiers appear in parentheses after man page names. GUI element This font denotes the names of graphical user interface (GUI) elements such

as windows, screens, dialog boxes, menus, toolbars, icons, buttons, boxes,

fields, and lists.

Product Support

SGI provides a comprehensive product support and maintenance program for its products, as follows:

• If you are in North America, contact the Technical Assistance Center at

+1 800 800 4SGI or contact your authorized service provider.

• If you are outside North America, contact the SGI subsidiary or authorized distributor in

your country.

007-4857-002 xxi

About This Guide

Reader Comments

If you have comments about the technical accuracy, content, or organization of this document, contact SGI. Be sure to include the title and document number of the manual with your comments. (Online, the document number is located in the front matter of the manual. In printed manuals, the document number is located at the bottom of each page.)

You can contact SGI in any of the following ways:

• Send e-mail to the following address: techpubs@sgi.com

• Contact your customer service representative and ask that an incident be filed in the SGI incident tracking system.

• Send mail to the following address: Technical Publications

SGI 1140 East Arques Avenue, M/S 50-1-946 Sunnyvale, California 94085

SGI values your comments and will respond to them promp tly.

xxii 007-4857-002

Chapter 1

1. System Installation

This chapter describes how to unpack and install your Altix 450 system. Specifically, the following topics are covered:

• “System Installation Overview” on page 2

• “Safety Precautions” on page 3

• “Installing a Rack” on page 6

• “Unpacking and Inspecting Modules” on page 22

• “Installing the IRUs in a Rack” on page 24

• “NUMAlink Cabling the System IRUs” on page 29

• “Cabling the System IRUs to a Power Source” on page 31

If you have arranged to have your system installed and set up by SGI professional services personnel, you can skip this chapter and go on to Chapter 2, “Operation Procedures”.

007-4857-002 1

1: System Installation

System Installation Overview

In this chapter, different installation instructions are provided for different types of system installations. Follow the instructions for your type of installation, as follows:

Note: It is assumed that you will be connecting a system console to your system in each installation case.

• If you have ordered a server system that is already rackmounted, which means that the modules that compose your system are also already cabled together, follow the instructions in these sections:

– “Safety Precautions” on page 3 – “Installing a Rack” on page 6 – “Cabling the System IRUs to a Power Source” on page 31 – “Connecting the System Console” on page 36

• If you have ordered un-racked IRU modules with the intention of rackmounting and cabling your own server system, follow the instructions in these sections:

– “Safety Precautions” on page 3 – “Installing a Rack” on page 6 – “Installing the IRUs in a Rack” on page 24 – “NUMAlink Cabling the System IRUs” on page 29 – “Cabling the System IRUs to a Power Source” on page 31 – “Connecting the System Console” on page 36

2 007-4857-002

Safety Precautions

• If you have ordered either a single- or dual-IRU system that you intend to install on a table

top, follow the instructions in these sections: – “Safety Precautions” on page 3 – “Unpacking and Inspecting Modules” on page 22 – “Installing the IRUs in a Rack” on page 24 – “Cabling the System IRUs to a Power Source” on page 31 – “Connecting the System Console” on page 36

• If you have your own rack, and have ordered system modules with the intention of

rackmounting and cabling your own system, follow the instructions in these sections: – “Safety Precautions” on page 3 – “Unpacking and Inspecting Modules” on page 22 – “Installing the IRUs in a Rack” on page 24 – “NUMAlink Cabling the System IRUs” on page 29 – “Cabling the System IRUs to a Power Source” on page 31

Safety Precautions

Before you install an Altix 450 system, you should familiarize yourself with the safety precautions discussed in the following subsections:

• “Hazard Statements” on page 4

• “ESD Precautions” on page 4

• “Safety Measures” on page 5

– “Connecting the System Console” on page 36

007-4857-002 3

1: System Installation

Hazard Statements

During the installation of your Altix 450 system, be alert for hazard advisory statements with icons, which signify the following:

• Caution indicates a potentially hazardous situation that, if not avoided, can result in minor or moderate injury. A caution statement also alerts you to unsafe practices that can result in equipment damage and/or data corruption. A caution message is accompanied by an icon as shown in the following example:

ESD Precautions

Caution:

• Warning indicates a potentially hazardous situation that, if not avoided, could result in death or serious injury. A warning message is accompanied by icon as shown in the following example:

Warning:

• Danger indicates an imminently hazardous situation that, if not avoided, will result in death or serious injury. A danger message is accompanied by the same icon as a warning.

Observe electrostatic discharge (ESD) precautions during the entire installation process to eliminate possible ESD damage to the equipment. Wear an SGI-approved wrist strap when you handle an ESD-sensitive device. Connect the wrist strap cord directly to earth ground.

Caution: Observe all ESD precautions. Failure to do so can result in damage to the equipment.

4 007-4857-002

Safety Measures

Safety Precautions

Observe the following safety measures when you install the system:

• Use caution when you remove the system from the shipping crate. Failure to handle the

system carefully can result in personal injury or property damage.

Caution: Ensure that the shipping crate is positioned close to its destination before you

unpack the crate.

W arning: Employ a minimum of two people to lift an IRU, to move the IRU(s) from

one location to another , and to install the IRU(s) in a rack. Otherwise, someone could be seriously injured. Note that IRUs should have blades and power supplies removed to make the unit easier to lift and position.

• Do not move the system while it is connected to power.

Warning: Keep fingers and conductive tools away from high-voltage areas. Failure

to follow these precautions will result in serious injury or death. Th e high-volt age ar eas of the system are indicated with high-voltage warning labels.

• Ensure that a qualified electrician has properly installed the power receptacles.

• Set all PDU circuit breakers to the OFF (

cord.

O) position before you plug in the system power

Warning: Use the following guidelines to prevent the rack from toppling over.

Otherwise, people could be seriously injured and/or equipment could be damaged.

• Follow these guidelines to prevent a rack from toppling over:

– Install all equipment in the lowest available position in the rack. – Use optional seismic tiedowns where appropriate. – Always site the unit on a level surface.

007-4857-002 5

1: System Installation

Installing a Rack

This section describes how to install a rack that already has your system IRUs rackmounted and cabled together. The following information is provided:

• “Preinstallation Activities” on page 6

• “Unloading and Moving System Equipment” on page 12

• “Removing a Short Rack from the Shipping Crate” on page 16

• “Unpacking and Installing the Tall Rack” on page 18

• “Positioning and Leveling Racks” on page 21

After you have completed installing your rack, proceed to “Cabling the System IRUs to a Power Source” on page 31 to continue your installation.

If your system components were not shipped in a rack, skip this section and proceed to “Unpacking and Inspecting Modules” on page 22 to begin your installation.

Preinstallation Activities

Perform the following preinstallation activities:

• Perform site verifications (see “Site Plan Verification” on page 6)

• Gather appropriate tools to complete the installation (see “Tools Required” on page 7)

• Ensure that the correct power receptacle is installed and properly wired (see “Power

Note: You can perform the preinstallation activities days or weeks before you receive your system.

Site Plan Verification

Ensure that all site requirements are met before your system arrives. If you have questions about the site requirements or you would like to order full-size floor templates for your site, contact a site planning representative by e-mail (site@sgi.com).

Receptacle Verification” on page 8)

6 007-4857-002

Tools Required

Installing a Rack

Table 1-1 lists the tools that you need to complete the installation.

Table 1-1 Installation Tools

Tool Part Number Purpose

13-mm wrench 7260744 Adjust the leveling pads. 13-mm socket (3/8-in. drive) 7260726 Remove bracket bolts from tall rack shipping crate. 19-mm socket (3/8-in. drive) 9470618 Remove bolts from short rack shipping crate. Extension, 6-in. (3/8-in. drive) 7260655 Used with ratchet and sockets. Ratchet, reversible (3/8-in. drive) 7260755 Used with extension and sockets. Level, 9-in. 9470556 Level the rack.

007-4857-002 7

1: System Installation

Power Receptacle Verification

Ensure that a qualified technician installs the correct power receptacles. Your Altix 450 system uses one or two single-phase power receptacles. For North American sites, the single-phase receptacle is a 30-amp, 200- to 240-volt receptacle that has two phase sockets and one ground socket. For international sites, the single-phase receptacle is a 32-amp, 200-volt receptacle that has one phase socket, one neutral socket, and one ground socket. Note that the ground pin is slightly longer than the other two pins on the North American plug.

Optional three-phase power distribution is available in systems mounted in 42U (tall) racks.

For North American sites, follow these steps to ensure that a single-phase power receptacle is properly wired:

1. Set the voltmeter to a high AC voltage range.

2. Check the voltage between socket X and socket Y (see Figure 1-1). The meter should read

3. Check the voltage between socket X and the ground socket. The meter should read

between 200 and 240 VAC.

approximately 120 VAC.

4. Check the voltage between socket Y and the ground socket. The meter should read approximately 120 VAC.

5. Check the voltage between the ground socket and an earth-ground location. The meter should read 0 VAC.

6. Change the voltmeter to a low-resistance setting.

7. Measure between the ground socket and an appropriate earth-ground location and ensure that resistance is less than 1 ohm.

8. Repeat steps 1 through 7 for any additional single-phase power receptacles.

Caution: If a voltage reading is incorrect, or if the resistance measured in step 7 is more than 1

ohm, contact a site-approved electrician. Do not proceed with the installation.

8 007-4857-002

Pole X

Socket Y

Installing a Rack

Power cord connector

Ground pole

Pole Y

Ground socket

Socket X

Receptacle

Figure 1-1 30-amp Single-phase Power Receptacle for North American Sites

007-4857-002 9

1: System Installation

For international sites, follow these steps to ensure that a single-phase power receptacle is properly wired:

1. Set the voltmeter to a high AC voltage range.

2. Check the voltage between socket 1 and socket 2 (see Figure 1-2). The meter should read between 200 and 240 VAC.

3. Check the voltage between socket 1 (line) and the ground socket. The meter should read between 200 and 240 VAC.

Note: The ground pin is slightly larger in diameter than the other two pins.

4. Check the voltage between socket 2 (neutral) and the ground socket. The meter should read approximately 0 VAC.

5. Check the voltage between the ground socket and an earth-ground location. The meter should read 0 VAC.

6. Change the voltmeter to a low-resistance setting.

7. Measure between the ground socket and an appropriate earth-ground location and ensure that resistance is less than 1 ohm.

8. Repeat steps 1 through 7 for any additional single-phase power receptacles.

Caution: If a voltage reading is incorrect, or if the resistance measured in step 7 is more than 1

10 007-4857-002

ohm, contact a site-approved electrician. Do not proceed with the installation.

Pin 1 (line)

Socket 1 (line)

Ground pin

Installing a Rack

Pin 2 (neutral)

Power cord connector

Ground socket

Socket 2 (neutral)

Receptacle

Figure 1-2 32-amp Single-phase Power Plug for International Sites

007-4857-002 11

1: System Installation

Unloading and Moving System Equipment

Your Altix 450 system arrives at the site in a pallet-mounted cardboard shipping crate.

For a short rack system, the documentation carton and the accessories carton are packed with the system. The documentation carton contains the system manuals as well as warranty and licensing information. The accessories carton contains the I/O, peripheral, and system cables, and any additional connectors or tools that are required for a specific configuration. Optional storage or monitors are shipped in a separate carton.

For a tall rack system, the system documentation; accessories; and other optional hardware arrive in separate cartons.

If you have purchased optional installation support from SGI service skip the following sections and go on to the next chapter.

This section describes how to unload and transport the rack system to its designated location, as follows:

• “Unloading the Equipment from the Truck” on page 12

• “Inspecting the Shipping Crate” on page 14

• “Transporting the Shipping Crate” on page 15

Unloading the Equipment from the Truck

If your loading dock is the same height as the transportation vehicle, use a pallet jack to unload the system from the transportation vehicle. The pallet jack should have 48-in. tines or forks. Follow any instructions that are printed on the packing crates.

If the loading dock is not the same height as the vehicle, you must provide a forklift or another approved method to unload the system. You can use a platform or ramp to obtain the desired level as long as the ramp incline does not exceed a ratio of one unit vertical to six units horizontal. For more information on site requirements, contact SGI site planning by e-mail (site@sgi.com).

Warning: Use two or more people to prevent computer equipment from rolling off the

transportation vehicle. Failure to do so could result in serious damage to the computer equipment.

12 007-4857-002

Installing a Rack

If your site does not have a loading dock, arrange for a forklift to unload the system from the transportation vehicle. Ensure that two or three people are available to help unload the equipment. Move all crates slowly and carefully.

Figure 1-3 shows the lift openings and dimensions of a tall rack shipping crate. This figure also shows where to position the pallet jack.

Side view

81.5

(2070.1 mm)

52.0

(1320.8 mm)

Pallet jack position

End view

(1066.8 mm)

Figure 1-3 Dimensions of Tall Rack Shipping Crate

007-4857-002 13

1: System Installation

(

Figure 1-4 shows the lift openings and dimensions of a short rack shipping crate. This figure also shows where to position the pallet jack.

1238 mm)

Pallet jack position

Figure 1-4 Dimensions of Short Rack Shipping Crate

Inspecting the Shipping Crate

After the system is unloaded from the truck, follow these steps before you unpack it:

1. Ensure that the crates and cartons arrive unopened.

2. Inspect the shipping crate for signs of external damage such as dents, holes, crushed corners,

3. Ensure that the tilt watch has not been tripped.

Side view

48.75

(1068 mm)

and water marks.

42.75

End view

29.25

(743 mm)

4. If the crate is damaged, file a damage claim with the carrier immediately. In addition, notify your local Customer Support Center (CSC) for any missing, incorrect, or damaged items. For CSC contact information, see http://www.sgi.com/support/supportcenters.html.

14 007-4857-002

Transporting the Shipping Crate

Use a pallet jack with forks that are 48 in. (122 cm) long or longer to transport the shipping crate to the designated location. See Figure 1-3 and Figure 1-4 for the crate dimensions and location to position the pallet jack.

If the crate does not fit through all access doors, you may need to partially disassemble the crate.

Caution: If the system shipping or storage environment is significantly colder than the

environment in which it will be installed [40 shipping crate for at least 24 hours at room temperature before you start the installation . This acclimation prevents damage to the equipment that could result from thermal shock and condensation.

Installing a Rack

ºF (22 ºC) or greater disparity], leave the rack in its

007-4857-002 15

1: System Installation

Removing a Short Rack from the Shipping Crate

W arning: Be caref ul when you unpack and move the short rack system. Use a pallet jack

to move the rack to the unpacking site. Ensure that the rack remains on a level surface and that the rack weight remains evenly distributed across the four casters. If you must lift the casters over an obstacle, such as a door threshold, use proper lifting tech niques and employ a minimum of two people.

Caution: Do not subject the rack to any unnecessary shocks or vibration while you unpack and

install the system.

See Figure 1-5 as you follow these steps; the numbered illustrations correspond to the numbered steps.

1. Ensure that the temperature of the rack is acclimated to the environment in which you are installing it and move the crate to the installation location.

2. Remove all external banding and lift the crate’s ramp away from the unit.

3. Remove the cardboard carton top and packing/insulation materials.

4. Prepare to move the rack off the crate using the following steps: a. Remove the two bolts that hold the pallet’s rear retention bracket in place. b. Remove the two bolts that hold the rear “gate” of the pallet in place. c. Remove the two bolts that hold the pallet’s front retention bracket in place.

5. Align the holes in the edge of the ramp with the holes at the rear of the crate and secure the ramp in place using the two bolts removed from the pallet’s rear gate.

6. Using two people, carefully pull the rack off the crate and roll it down the ramp.

Warning: The maximum weight of the short rack is 750 lbs (340 kg). Use caution when

you roll the rack down the ramp.

16 007-4857-002

Installing a Rack

4a 4b

Rear mounting

bracket

Ramp

Cardboard

cover

Foam

cushion

Front mounting

bracket

Figure 1-5 Removing a Short Rack from the Shipping Crate

007-4857-002 17

1: System Installation

Unpacking and Installing the Tall Rack

W arning: In its maximu m configuration, a (42U) tall rack system weighs approximately

1,450 lb (658 kg). Use caution when you unpack and move this rack. A pallet jack is needed to move the rack prior to unpacking. Ensure that the rack remains on a level surface and that the rack weight remains evenly distributed across the four casters.

To unpack a tall rack, you will need the following tools:

• Extension, 6-inch (15.3 cm), 3/8-in. (13mm) drive

• 13-mm standard 3/8-in. drive socket

• Ratchet, reversible, 3/8-in. (13mm) drive

See Figure 1-6 as you follow these steps; the numbered illustrations correspond to the numbered steps. Ensure that the temperature of the rack is acclimated to the installation environment and that the system crate is stable, and in an upright position.

1. Use a sharp cutting tool to remove the bands th at secure the cardboard crate, then remove the outer packing materials.

2. Remove the cover bag and bungee cushion from the rack.

3. Locate and remove the 8 bolts that secure the wooden frame pieces to the base.

4. The rack’s front and rear doors must be removed before you can remove the rack’s shippi ng runners.

5. Remove and discard the foam plugs from either end of the rack’s base. At this time you will need to place a pallet jack under the rack and lift it until the shipping runners are clear of the floor.

6. Undo and remove the two bolts on the shipping runners at either end of the rack and set the runners aside.

7. Use the pallet jack to lower the rack to the floor and use at least two people to roll the rack to the installation site.

Warning: Personal injury and system damage could result if the rack becomes

unbalanced or gains too much momentum when rolled.

Reattach the doors to the front and rear before you move the rack to its designated location (see Figure 1-7 on page 20).

18 007-4857-002

Installing a Rack

Bungee cushion

COMPACT COMPACT

COMPACT

Foam plug

COMPACT COMPACT

COMPACT

COMPACT COMPACT

Figure 1-6 Removing a Tall Rack from the Shipping Crate

007-4857-002 19

1: System Installation

COMPACT COMPACT

Figure 1-7 Reattaching the Tall Rack Doors

20 007-4857-002

Positioning and Leveling Racks

Caution: T o avoid ESD damage to the electronic components, be sure to position the racks before

you remove the ESD bags that cover the rack assemblies.

To position and level your system racks, follow these steps:

1. Grasp the rear of each rack and roll the rack to its designated location.

2. Remove the ESD bags.

3. Adjust the leveling bolts of a rack, as shown in Figure 1-8, until the rack is level.

Installing a Rack

Figure 1-8 Leveling Bolts

4. If you are installing additional racks, repeat until all of the racks are level.

5. Ensure that the circuit breakers on the power distribution un its are in the OFF (

Then connect the power cords to grounded power outlets. Plugging in the power cords grounds the racks.

6. It is recommended you secure all racks with optional seismic tie-downs if you are installing

the system in an earthquake zone.

Note: Ta ll and short racks have four threaded holes that are located at the bottom of the rack (see Figure 1 -9). Use these holes to secure the seismic tie-downs. Note that SGI does not supply the seismic tie-downs.

007-4857-002 21

O) position.

1: System Installation

op view of short rack

Top view of tall rack

Seismic tie-down attachment poin Casters

Leveling pads

Figure 1-9 Seismic Tie-down Attachment Points

Unpacking and Inspecting Modules

This section describes how to unpack and inspect individual IRUs. Follow these instructions if you plan to install individual modules in a rack or on a table top.

Before unpacking your IRUs, inspect the packaging container for evidence of mishandling during transit. If the packaging container is damaged, photograph it for reference. After you remove the contents, keep the damaged container and the packing materials.

Remove the IRU(s) from the packaging container and ensure that all accessories are included. Inspect the IRU(s) and accessories for damage. If the contents appear damaged, file a damage claim with the carrier immediately. In addition, notify your local SGI Customer Support Center (CSC) for any missing, incorrect, or damaged items. For CSC contact information, see http://www.sgi.com/support/supportcenters.html.

22 007-4857-002

Note: When using the IRU as a stand-alone unit (set on a desk or table top) you must secure the IRU and its fan module together with connection plates. IRUs mounted in a rack are held in place by the screws at the front and rear of the unit.

Unpacking and Inspecting Modules

Firmly connect the base module of the IRU to the fan module using these steps:

1. Align the rear connectors of the fan module with the rear of the IRU base module and

carefully slide it in until it fully joins together.

2. Align the right side joining plate screw holes with the eight holes in the two adjacent

modules. See Figure 1-10 for an example.

3. Insert and tighten the eight retention screws that hold the connection plate in place.

4. Repeat the process for mounting the joining plate on the left side of the IRU.

COMPACT

i n si de

inside

Figure 1-10 IRU Connection Plate for Desktop Systems

007-4857-002 23

1: System Installation

Installing the IRUs in a Rack

This section describes how to install the IRUs that compose an Altix 450 system in a rack. This section also describes how to remove the IRUs from a rack. You can use the instructions to rackmount one or more IRUs in a 20U (short) or 42U (tall) Altix rack.

Caution: Always use at least two people when moving, installing, or removing a rackmounted

Determining Space Requirement s

Altix 450 IRU.

This rackmounting information is found in the following sections:

• “Determining Space Requirements” on page 24

• “Determining Where to Attach Rails in a Rack” on page 25

• “Rackmounting IRUs with Shelf Rails” on page 26

Table 1-2 specifies the space requirements when rackmounting an IRU in a 19-inch rack.

Table 1-2 IRU Space Requirements

Height (5U) 8.68 inches (22 cm) Width 17.5 inches (44.45 cm) Depth 32.5 inches (82.5 cm) with blower attached Weight ~130 lb (59 kg) max config Required front clearance for IRU 8.25 in. (20.96 cm) Required rear clearance for IRU 10 in. (25.40 cm) Required side clearance for IRU No clearance requirement Required front clearance for rack 48 in. (122 cm) Required rear clearance for rack 48 in. (122 cm)

a. Weight will vary slightly depending on the number of blades installed and the amount of DIMMs installed, and

on whether you have one or two disk drives in the IRU.

24 007-4857-002

Determining Where to Attach Rails in a Rack

The base IRU, and any additional IRUs, each require five units (5U) of vertical space within the rack (one unit is equivalent to 1.75 inches [44.5 mm]). To determine where you should install the unit’s shelf rails in the rack, you must count mounting holes. Each U contains three mounting holes; therefore, in the 5U of space that the module occupies, there are 15 mounting holes. The bottom hole of the 5U space is hole 1. The top mounting hole in the 5U space is hole 15. See Figure 1-11.

Note: An IRU in the rack is identified by the lowest U number that it occupies. For example, in Figure 1-11 the IRU’s bottom is in U1 (the first unit within the rack).

Installing the IRUs in a Rack

6th hole

inside

COMPACT

Figure 1-11 Mounting Hole Pattern of Rack Rails

To determine how many mounting holes you must count, use the following formula: 3x(the lowest U number that the module will occupy) - 2. For example, when you want to install the module in locations U6 through U11, count 16 mounting holes (3 x 6 - 2) starting from the bottom of the rack. The 16th hole is the first mounting hole of U6.

007-4857-002 25

1: System Installation

Rackmounting IRUs with Shelf Rails

This section describes how to rackmount an IRU using shelf rails. The shelf rails, which are shipped with the module, support the module within the rack. To attach the shelf rails to the rack, follow these steps:

1. Locate the slot location in the rack in which you will install the IRU. See “Determining Where to Attach Rails in a Rack” on page 25 if you are uncertai n.

2. In the slot location from step 1, position the left shelf rail at a 90-degree angle to the front and rear rack rails (see Figure 1-12).

3. Insert the top-front bracket tab into one of the narrow slots in the front rack rail.

4. Insert the top-rear bracket tab into the narrow slot in the rear rack rail.

Note: Verify that the tabs are the same height to ensure that the shelf is level in the rack.

5. Lower the bottom of the shelf rail until its two lower tabs engage with the wide slots in the front and rear rack rails. The shelf rail is held in by gravity and the tabs.

6. Repeat these steps for the right shelf rail.

Figure 1-12 Installing the Shelf Rails in the Rack

26 007-4857-002

Installing the IRUs in a Rack

Note: Even with the blades, power supplies and fan module disconnected, the IRU’s base unit is heavy. Always use two people when installing it into a rack.

7. Working from the front of the rack, set the rear edge of the IRU module on the shelf rails.

Then slide the module into the rack until the module ears are snug against the front rack rails. See Figure 1-13.

8. Fasten the module to the front rails of the rack with the screws provided. See Figure 1-13.

9. Working from the rear of the rack, align the rear connectors of the fan module with the rear

of the IRU base module and carefully slide it in until it fully connects together.

10. Fasten the fan module to the rear rails of the rack with the screws provided.

COMPACT

inside

10-32 x 1/2-inch screws

Figure 1-13 Securing the Module to the Front of the Rack

007-4857-002 27

1: System Installation

Removing a Rackmounted IRU Module

To remove an IRU that is already mounted in a rack, follow these steps:

1. Power off the module. For instructions on how to power off the module, see “Powering the System On and Off” on page 38.

2. Disconnect all of the cables at the front of the module.

3. Removing the system blades and power supplies from the IRU is recommended to decrease the overall weight of the unit. Be sure to place any system blades in a static-free area.

Warning: Components may be hot. To avoid injury, allow the components to cool for

approximately five minutes before you proceed with these instructions.

4. Remove the screws that secure the module to the front rails of the rack.

5. Remove the screws that secure the IRU at the rear of the rack.

Caution: Always use at least two people when installing or removing a rackmounted Altix 450

IRU.

6. Use a minimum of two people to slide the unit out the front of the rack. Both lifters need to place one hand securely under the IRU as it slides out to support the weight of the unit as it clears the rack.

7. Place the IRU base module on a flat, stable surface.

8. Remove the IRU’s fan module from the rear of the rack and place it nearby.

28 007-4857-002

NUMAlink Cabling the System IRUs

Some configurations of an Altix 450 system will consist of either two or more IRUs. This section describes how to cable together these multiple units. If your system was shipped with all the cables pre-installed, go on to the next section.

Note: If your system is made up of a single module, you can skip this section and proceed to “Cabling the System IRUs to a Power Source” on page 31 for instructions to connect your system to power.

The primary IRU module (housing the system disk), connects to additional Altix 450 IRUs via the module’s front NUMAlink connectors. Figure 1-14 shows an example.

Note: Before you connect two Altix 450 IRUs to each other, you must assign unique system ID numbers to the compute modules, as follows:

1. Designate one of the IRU modules as the master server and the other IRU chassis as the

slave.

NUMAlink Cabling the System IRUs

2. Connect a console to the master IRU (see “Connecting the System Console” on page 36),

and then power on the IRU (see “Powering the System On and Off” on page 38).

3. Use the <slotnumber> L1 command to set the secondary IRU’s ID number to a higher

number than the primary IRU’s ID number . For example, if the master IRU unit’ s L1 prompt indicates that its ID number is 01 ( should be 02 or higher (

001c01-L1>brick slot 02 brick slot set to 02.

001c02-L1>).

001c01-L1>), then the subsidiary IRU ID number

See the SGI L1 and L2 Controller Software User’s Guide (007-3938-00x) for more information on L1 commands.

007-4857-002 29

1: System Installation

i ns i de

COMPACT COMPACT

i ns i de

Figure 1-14 NUMAlink Cabling an IRU to an Additional IRU

To connect two Altix 450 compute modules to each other with NUMAlink cables, follow these steps:

1. Power off both compute modules (see “Powering the System On and Off” on page 38).

2. Connect the NUMAlink cables to the NUMAlink connectors on both IRU modules.

3. Connect a system console to the master IRU (the one with the system disk), as described in “Connecting the System Console” on page 36.

4. Power on both IRUs as described in “Powering the System On and Off” on page 38.

30 007-4857-002

The console output should indicate that the processors and memory on both IRU modules have been discovered. If the secondary IRU module has not been discovered, use the reset L1 command to perform a reset of the system.

Note: Use the network L1 command to set the network mode. (See the SGI L1 and L2 Controller Software User’s Guide (007-3938-00x) for more information on L1 commands.)

Cabling the System IRUs to a Power Source

This section describes how to make the following connections to a power source:

• “Connecting an IRU Module to a Power Source (No Rack)” on page 31

• “Connecting System Modules to a Rack Power Source” on page 34

Connecting an IRU Module to a Power Source (No Rack)

Cabling the System IRUs to a Power Source

If you are operating your system on a table top, connect the IRU(s) to an approved electric receptacle. See Figure 1-15 for the location of the IRU’s power connectors.

007-4857-002 31

1: System Installation

inside

Figure 1-15 Power-Supply Connectors on the IRU Module

Figure 1-16 shows the connecting of a dual-IRU system located on either a tabletop or desktop, to a power source.

32 007-4857-002

ower

ource

Power distribution unit (PDU)

Cabling the System IRUs to a Power Source

COMPACT

i n s i d

T M

COMPACT

in s ide

T M

Figure 1-16 Connecting an Unracked IRU System to a Power Source

007-4857-002 33

1: System Installation

P s

Connecting System Modules to a Rack Power Source

If your Altix 450 system is shipped from SGI in a rack, there will be at least one power distribution unit (PDU) installed in the rack. Note that each single-phase PDU can support a maximum of two Altix 450 IRUs. Plug both power cords from each IRU into the PDU(s) installed in your short or tall rack.

Figure 1-17 shows an example of cabling the system modules that reside in a rack to a PDU.

Power distribution unit (PDU)

COMPACT

i nsid

COMPACT

i nside

ower

ource

Components within rack

Figure 1-17 Connecting Multiple Modules to a PDU

An optional three-phase PDU (used in 42U system racks only) has 18 outlet connections, of which up to 16 may be needed to power eight IRUs. Each three-phase rack system houses a single PDU.

34 007-4857-002

Cabling the System IRUs to a Power Source

n )

Figure 1-18 shows an example of the optional three-phase PDU available in tall racks only.

Power distributio unit (PDU

Figure 1-18 Optional Three-Phase PDU Example

007-4857-002 35

1: System Installation

Connecting the System Console

This section explains how to attach and establish a connection between a system console and an Altix 450 system. The system console enables you to perform the following activities:

• Start a Linux console session to set up networking and other configuration parameters.

• Monitor your system by reading the status and error message information that the L1 controller generates.

• Enter L1 controller commands to monitor or change particular system functions. You can, for example, monitor the speed of fans for a particular module. See the SGI L1 and L2 Controller Software User’s Guide (007-3938-00x) for descriptions of the L1 controller commands that you can use.

• Power-on or power-off your system.

To attach a PC to your server system, connect a null modem serial cable between the PC and the Console port of the IRU (see Figure 1-19). To establish a connection between your system and the system console (PC), follow these steps:

1. Ensure that the L1 controller of the base compute module is powered on. When the base compute module is connected to a power source, the L1 controller should be powered on.

Note: When powered on, the L1 controller displays L1 running. If it does not display L1 running, check the connection between the base compute module and the power source. In

addition, ensure that the circuit breaker of the power source is on.

2. Power on the system console.

3. Ensure that the system console has the following network settings:

• Baud 38400

• No parity

• 8 data bits

• 1 stop bit

• Hardware flow control on (RTS/CTS)

4. Type cu -l ttys0 -S38400 for a Linux based console, and then press Enter to display the L1 prompt. If the console uses another operating system, start a terminal emulation program as instructed in the console’s user documentation.

36 007-4857-002

Console port

Reset button

Connecting the System Console

inside

thernet port

L1 controller

panel

Figure 1-19 Location of the L1 Console Port

007-4857-002 37

1: System Installation

Operating Your System

This section describes how to operate your Altix 450 system, as follows:

• “Powering the System On and Off” on page 38

• “L1 Front Panel Controls” on page 44

• “Operating the L1 Controller” on page 45

Powering the System On and Off

This section describes how to power on and power off your system (or an individual IRU) at a system console, or manually, as follows:

• “Preparing to Power On” on page 38

• “Powering Off at the System Console” on page 42

• “Powering On Manually or With L1 Console” on page 41

• “Powering Off at the System Console” on page 42

• “Powering Off Manually” on page 43

Preparing to Power On

To prepare to power on your system, follow these steps:

1. Confirm that all the cable connections between the IRUs and other system items are connected securely . If necessary, see “NUMAlink Cabling the System IRUs” on page 29 for information.

2. Confirm that all the power cables are plugged in securely to a power source. If necessary, see “Cabling the System IRUs to a Power Source” on page 31 for information.

Note: You must manually power on any optional storage modules by placing the power switch(es) in the ON (

I) position. Note that you should turn any optional storage module on before

starting the operating system.

38 007-4857-002

Powering On With Optional System Console and L1/L2 Controller

To power on and boot your system using the system console, follow these steps:

1. If not yet done so, connect a system console to the server system as described in

“Connecting the System Console” on page 36.

2. If the monitors and other peripherals are equipped with voltage select switches, verify that

they are set for the appropriate AC voltage and plug them in. Note that they are normally plugged into sources outside the rack system.

3. Turn on the on/off circuit breaker switchs of the PDU as shown in Figure 1-20.

Power distribution unit (PDU)

Operating Your System

On/off switche

Power source

Figure 1-20 PDU Circuit Breakers Example

007-4857-002 39

1: System Installation

4. Verify that all of the L1 controllers display “L1 run ning.” If any L1 controllers are not running, contact your SGI system support engineer (SSE).

5. At the system console, access the L2 controller by entering the following command:

$> /stand/sysco/bin/12term

6. From the L2 prompt, display the system configuration by entering the following command:

L2> cfg

This command lists the IRUs that the L2 controller detects in the system and their system controller addresses. If a IRU that you want to power on does not appear in the list, it will not power on. Ensure that the IRU’s L1 controller is running and that the IRU is cabled properly.

7. From the L2 prompt (L2>), power on an individual IRU by entering the following command. (If you want to power on the entire system, proceed to Step 8.)

L2> r <rack#> s <slot#> pwr u

For example, to power on a IRU in rack 1, slot 18, enter the following command:

L2> r 1 s 18 pwr u

The slot number is the unit number of the IRU within the rack. If you want to power on two selected IRUs of a rack at the same time, you must enter the

rack number followed by the slot numbers of the IRUs that you want to power on. For example, to power on the IRUs in slots 16 and 21, enter the following command:

L2> r 1 s 16,21 pwr u

8. If you want to power on the entire system, enter the following command:

L2> pwr u

(The default setting for the pwr u command is all racks and slots.)

9. When the L2 prompt appears, you will not see the output that is produced during the power-on procedure unless you redirect the keyboard input from the L2 controller to the normal console by typing control d.

L2> ctrl d

40 007-4857-002

Powering On Manually or With L1 Console

If your system does not have a a local system console, you can manually power on and off the server by using the power button(s).

To power on your system or an individual IRU manually, follow these steps:

1. If the monitors and other peripherals are equipped with voltage select switches, verify that

they are set for the appropriate AC voltage and plug them in. Note that they are normally plugged into power sources outside the rack system.

2. Turn the circuit breaker switchs of the PDU to the on position, see Figure 1-20 on page 39.

3. Press the power buttons (see Figure 1-21) on each of the IRUs that you want to power on.

Note that any optional storage modules should be powered up first.

inside

Operating Your System

1 controller

display

Power butto with LED

Figure 1-21 Location of the Power Button

T o boot the system from the L1 interface you must have a console connected to the system via the console port. Use the following commands depending on system type:

1. For a system that contains one IRU, enter the following command at the L1 prompt to power

on the server: 001c01-L1> power up

2. For a system with multiple IRUs, enter the following command at the L1 prompt:

001c01-L1> * power up

007-4857-002 41

1: System Installation

Powering Off at the System Console

To power off your system using the system console, follow these steps:

Caution: If you power off the system before you halt the operating system, you can lose data.

1. Shut down the operating system by entering the following command:

# init 0

2. T o access the L2 prompt, direct the keyboard input to the L2 controller by entering Ctrl+T.

$> ctrl t

3. From the L2 prompt (L2>), power off an individual IRU by entering the following command. (If you want to power off the entire system, proceed to Step 4.)

L2> r <rack#> s <slot#> pwr d

For example, to power off the IRU in rack 1, slot 16, enter the following command:

L2> r 1 s 16 pwr d

The slot number is the unit number of the IRU within the rack. If you want to power off more than one IRU in the rack at the same time, enter the rack

number followed by the slot numbers of the IRUs that you want to power off. For example, to power off the IRUs in slots 16 and 21, enter the following command:

L2> r 1 s 16,21 pwr d

4. If you want to power off all of the IRUs within the rack, enter the following command:

L2> pwr d

(The default setting for the pwr d command is all racks and slots.) “Powered Down” appears on the L1 display when the IRU is powered off. The L1 controller

is still powered on.

5. T o completely power down an L1 controller, unplug the IRU’s power cords from the PDU or two front power supplies.

42 007-4857-002

Powering Off Manually

Operating Your System

To power off your system manually, follow these steps:

Caution: If you power off the system before you halt the operating system, you can lose data.

1. Shut down the operating system by entering the following command:

# init 0

2. Press the power buttons or power switches on each of the IRUs that you want to power off.

You may power off the IRUs in any order:

• To power of f the optional storage units, press the power button(s) on its rear panel to the OFF (

O) position.

• To power off the IRUs, press the power button with the LED on the front panel of each unit (see Figure 1-21 on page 41).

007-4857-002 43

1: System Installation

L1 Front Panel Controls

The L1 front panel of the Altix 450 IRU provides the following control features, as shown in Figure 1-22:

• Status LEDs. The front panel has the following LEDs:

• Power button. Normally you power on the system from the system console. Alternatively,

• Reset button. Actuate this button to reset the internal processors and ASICs. Th e reset will

• NMI button. Actuate the NMI (non-maskable interrupt) button to reset the internal

• Power button LED. This LED illuminates green when the internal components are on.

• Service-required LED. This LED illuminates yellow to indicate that an item is not functioning properly (for example, a fan is off), but the IRU is operating.

• Failure LED. This LED illuminates red to indicate that a failure has occurred and the module is down.

you can press this button to power on the module.

cause a memory loss. (To perform a reset without losing memory, see the NMI button information that follows.)

processor(s) and ASICs, and write register data and memory to the /var/adm/crash file.

Use a paperclip or other small object to access the “pin-hole” actuators on the front panel.

L1 controller display

inside

ailure LED

Service-required LED

Power button

with LED

Reset

Figure 1-22 L1 Front Panel Functions

44 007-4857-002

NMI

Note: If your system includes an optional storage module, see the user’s guide that comes with the product for operating instructions.

Operating the L1 Controller

The L1 controller operates in one of the two following modes:

Operating Your System

• L1 mode. The L1 prompt (

001c01-L1>) is visible and all input is directed to the L1

command processor.

• Console mode from L1. Output from the system is visible and all input is directed to the system.

The L1 controller is ready to accept commands when you see a prompt of the following form:

001c01-L1>

See the SGI L1 and L2 Controller Software User’s Guide (007-3938-00x) for a detailed list of L1 commands.

To enter console mode, press Ctrl+D at the L1 prompt, as follows:

001c01-L1> Ctrl+D entering console mode 001c01 console, <CTRL-T> to escape to L1 . <system output appears here> .

To return to L1 mode, press Ctrl+T, as follows:

Ctrl+T

escaping to L1 system controller 001c01-L1>

At this point, you can enter any L1 command. When the command completes execution, the L1 returns to console mode:

re-entering console mode 001c01 console, <CTRL-T> to escape to L1

T o permanently engage the L1 mode, press Ctrl+T and then enter the l1 command, as follows:

Ctrl+T

007-4857-002 45

1: System Installation

escaping to L1 system controller 001c01-L1> l1 L1 command processor engaged, <CTRL-D> for console mode. 001c01-L1>

46 007-4857-002

Chapter 2

2. Operation Procedures

This chapter explains how to operate your new system in the following sections:

• “Precautions” on page 47

• “System Controller Network” on page 49

• “Powering the System On and Off” on page 51

• “Using Embedded Support Partner (ESP)” on page 56

• “Monitoring Your Server” on pa ge 57

• “Installing Optional Components” on page 59

Precautions

Before operating your system, familiarize yourself with the safety information in the following sections:

• “ESD Precaution” on page 47

• “Safety Precautions” on page 48

ESD Precaution

Caution: Observe all ESD precautions. Failure to do so can result in damage to the equipment.

W ear an SGI-approved wrist strap when you handle an ESD-sensitive device to eliminate possible ESD damage to equipment. Connect the wrist strap cord directly to earth ground.

007-4857-002 47

2: Operation Procedures

Safety Precautions

Warning: Before operating or servicing any part of this product, read the “Safety

Information” on page 16 5.

Danger: Keep fingers and conductive tools away from high-voltage areas. Failure to

follow these precautions will result in serious injury or death. The high-voltage areas of the system are indicated with high-voltage warning labels.

Caution: Power off the system only after the system software has been shut down in an orderly

manner. If you power of f the system before you halt the operating system, data may be corrupted.

W arning: If a lithium battery is installed in your system as a soldered part, only qualified

SGI service personnel should replace this lithium battery. For a battery of another type, replace it only with the same type or an equivalent type recommended by the battery manufacturer, or an explosion could occur. Discard used batteries according to the manufacturer’s instructions.

48 007-4857-002

System Controller Network

All Altix 450 enclosures contain an embedded microprocessor board and display assembly known as the system controller. This microprocessor runs an embedded version of the Linux operating system. The system controller runs off standby power and is running as long as the enclosure is connected to an active power source.

There are two primary applications that run on the system controller. The L1, or Level 1 system controller, is an application that provides control and monitoring functionality for each individual rack unit enclosure (IRU), and communication to other L1s in adjacent enclosures connected via NUMALink 4 cables. The L1 is always resident.

The console mode Level 2 (L2) system controller is an application that provides control over multiple L1s and communication to other L2s. The L2 is resident and active when the enclosure is connected by an Ethernet connection to a Local Area Network (LAN).

The system controller network provides the following functionality:

• Powering the entire system on and off.

• Powering individual IRUs on and off.

System Controller Network

• Monitoring the environmental state of the system

• Viewing the system’s status and error message information generated and displayed by the SGI system’s L1 controller.

• Enter L1 controller commands to monitor or change particular system functions. You can, for example, monitor the speed of fans for a particular individual rack unit (IRU) enclosure. See the SGI L1 and L2 Controller Software User’s Guide for a complete list of commands.

• Provides access to the system OS console allowing you to run diagnostics and boot the system.

007-4857-002 49

2: Operation Procedures

Console Connections

A console is defined as a connection to the system that provides access to the system controller network. A console can be a personal computer (Ethernet connection) or a dumb terminal (serial connection).

The Altix 450 series supports multiple types of console connections:

• An ethernet connection on the system control board of an IRU - an RJ45 connection is typically labeled “L2 host”

• A serial connection to the serial console port on the system control board - a DB9 connector typically labeled “Console”

• A 2D video card installed in the base I/O blade (not available with RHEL 5) - this low-profile PCI video interface card requires use of a USB keyboard and mouse

Connecting to the System Controller

The ethernet connection is generally the preferred method of accessing the system console. Depending on the size of the system, there will be one or more ethernet connections utilized.

When an enclosure is connected to the LAN, the system controller will start the ethernet interface with either the pre-assigned IP address or will attempt to acquire an IP address via DHCP, and will then spawn the L2 application. See the section on setting up an IP address for further details. Once the system controller is connected to the LAN, it can be accessed via a simple telnet session. The serial connection is used to communicate directly with the L1 system controller . This connection is typically used for service purposes or for system controller and system console access in small systems where an ethernet connection is not available.

Caution: An L1/L2 network connection must be kept on a private and physically secure network connection. The System Controller does not provide software security, authentication, or safeguards against malicious or careless users. The consequences of unauthorized access to the System Controllers are potential system unavailability, system reconfiguration, and detailed access to every aspect of the system. The recommended configuration for the L1/L2 network connection is a separate, private and physically secure network attachment to an equally separate and physically secure PC. Any Linux system hosting L3 software must be administered to prevent unauthorized access.Optional L3 software is a package that runs on a Linux based PC. It is mainly used by service personnel as a diagnostic aid.

50 007-4857-002

Communicating with the System Controller

Once a connection to the console is established, the user will be presented with either an L2 prompt (ethernet connection), known as L2 mode, or and L1 prompt (serial connectio n), known as L1 mode. From either of these prompts various system controller commands can be entered.

001c01-L1> <l1 command>

olympic-101-L2> <l2 command>

To access the system console, known as console mode, a control-d is entered at either the L1 prompt or the L2 prompt. To return to or escape back to the L1 or L2 mode, a control-t is entered. This escape is only temporary and you will be returned to console mode once the “Return/Enter” key is pressed. To re-engage L1 or L2 mode, enter control-t followed by either “l1” or “l2” depending on the original mode. System control commands are always entered in lower case unless otherwise specified.

Powering the System On and Off

Communicating with the System Controller

This section explains how to power on and power off individual rack units, or your entire Altix 450 system, as follows:

• “Preparing to Power On” on page 52

• “Powering On and Off at the Console” on page 54

For servers with a system console, you can power on and power off individual IRUs or the entire system at the system console.

You may also monitor and manage your server with tools such as VACM, Console Manager, and PCP. For details, see the documentation for the particular too l.

The Embedded Support Partner (ESP) program enables you and your SGI system support engineer (SSE) to monitor your server remotely and resolve issues before they become problems. For details on this program, see “Using Embedded Support Partner (ESP)” on page 56.

007-4857-002 51

2: Operation Procedures

Preparing to Power On

To prepare to power on your system, follow these steps:

1. Check to ensure that the cabling between the rack’s power distribution unit (PDU) and the wall power-plug receptacle is secure.

2. For each individual IRU that you want to power on, make sure that the power cables are plugged into all the IRU power supplies correctly, as shown in Figure 2-1. Setting the circuit breakers on the PDUs to the “On” position will apply power to the IRU and will start the system controller(s) in the IRUs. Note that the system controller in each IRU stays powered on as long as there is power coming into the unit.

inside

Figure 2-1 IRU Power Supply Cable Location Example

3. If you plan to power on a server that includes optional mass storage enclosures, make sure that the power switch on the rear of each PSU/cooling module (one or two per enclosure) is in the

1 (on) position.

4. Make sure that all PDU circuit breaker switches (see the example in Figure 2-2 on page 53) are turned on to provide power to the server system when it is switched on.

52 007-4857-002

Power

source

Powering the System On and Off

Power distribution unit (PDU)

On/off switche

Figure 2-2 Single-Phase Rack PDU Circuit Breaker Switches

007-4857-002 53

2: Operation Procedures

Powering On and Off at the Console

The power-on and of f procedure at a console varies with your server setup, as follows:

• If you have a console connected to a server with a serial interface, you can toggle between L1 and console mode. This enables you to power on your server with L1 commands and view the activity by changing to the console mode.

For detailed instructions on using a system console using the L1 mode, see “Operating the L1” on page 84.

• If you have a system console connected to a server with an ethernet interface, you can toggle between L2 and console mode, and power on your server with L2 controller commands, and monitor the power-on activity by changing to the console mode.

Powering On and Off from the L1

Commands issued at the L1 prompt typically only affect the local IRU. The following sections describe how to power on and power off your system in L1 mode.

The L1 controller display, located on the front of each IRU, should display

L1 running once the

power-on procedure starts (storage modules do not have L1s). The prompt on your console screen shows the rack and slot number of the IRU to which you have connected your console.

Note: If you have a problem while powering on and an error message appears on your console display, see “L1 Controller Error Messages” on page 145 to learn what the error message indicates and how to resolve the problem.

If you want to power on the IRU (001c01 in our example) indicated in the prompt, enter the following command.

001c01-L1> power up

To power off:

001c01-L1> power down

If you are attempting to power on or power off a system with multiple IRUs from the L1 prompt, you will need to prepend the command with an asterisk as follows:

oo1c01-L1> * power up or

54 007-4857-002

Powering the System On and Off

oo1c01-L1> * power down

(* indicates all)

From the L1 prompt, display the system configuration information by entering the following command:

001c01-L1> config :0 001c01 L0C :2 002r01 L1H :4 002r05 L3H 001c01-L1>

In L1 mode, you can obtain only limited information about the system configuration. An IRU has information about its internal blades, and also when other IRUs are NUMAlink attached to the IRU, information about those IRUs.

Note: Verify that the power LED on the L1 display turns on and lights green and that your controllers display that the system is powered on for each segment of the procedure, which indicates that the power-on procedure is proceeding properly. If you have a problem while powering on and an error message appears on the L1 controller, or the system console, see your online log files and the information in “L1 Controller Error Messages” on page 145 to learn what the error message indicates and how to best resolve the problem.

007-4857-002 55

2: Operation Procedures

Powering-On and Off From the L2

Commands issued at the L2 prompt are typically sent to all NUMAlink connected IRU enclosures in the system.

To power on a system:

system-101-L2> power up

To power off a system:

system-101-L2> power down

Using Embedded Support Partner (ESP)

Embedded Support Partner (ESP) automatically detects system conditions that indicate potential future problems and then notifies the appropriate personnel. This enables you and SGI system support engineers (SSEs) to proactively support systems and resolve issues before they develop into actual failures.

ESP enables users to monitor one or more systems at a site from a local or remote connection. ESP can perform the following functions:

• Monitor the system configuration, events, performance, and availability.

• Notify SSEs when specific events occur.

• Generate reports.

ESP also supports the following:

• Remote support and on-site troubleshooting.

• System group management, which enables you to manage an entire group of systems from a single system.

For additional information on this and other available monitoring services, see the section “SGI Electronic Support” in Chapter 7.

56 007-4857-002

Monitoring Your Server

You can monitor you r Alt ix 450 server from the following sources:

• On the L1 controller’s display at the front of the IRU as shown in Figure 2-3, you can

monitor system operational status. For example, you can monitor error messages that warn of power or temperature values that are out of tolerance.

• You can connect and monitor the system via an Ethernet port adapter. You will need to

connect either a local or remote workstation to monitor the system via Ethernet. These console connections enable you to view the status and error messages generated by

the L1/L2 controllers in your Altix 450 rack. You can also use these consoles to input L1/L2 commands to manage and monitor your system.

• A DVI /VGA video card allows you to interact directly with the system, though not directly

with the system’s L1/L2 interface. See “2D Graphics Video Interface” on page 58 for more information. SLES 10 or later is required for this card. RHEL 5 is not supported.

Monitoring Your Server

L1 controller

panel

ort

PCI-X 1 PCI-X 0

DVDROM

HDD0 HDD1

ETH0 ETH1

P0PF N0N1

F1F2 F3 F4

A1A2 A3A4

SAS 4X USB0 USB1 USB2 USB3

inside

Po G

Po H

Power

Compute/memory blades

Router

board

IA blade

System control

board

supplies

Router

board

Figure 2-3 IRU L1 Controller Display Location Example

007-4857-002 57

2: Operation Procedures

2D Graphics Video Interface

If your system was ordered in January 2007 or later, it may come equipped with an optional 2D graphics board interface (supported with SLES 10 or later system software). This low-profile PCI interface card is installed in the base I/O blade in the Altix system (see Figure 2-4 on page 59). Note that this blade is commonly referred to as the IA2 blade. One 2D card is supported per system or partition; no audio function is supported. The 2D video card has the following features:

• 64 MB DDR graphics memory

• DMS-59 display connector for two display support (no synchronization between monitors)

• Simultaneous display to DVI and VGA monitors using an adapter

• Analog resolution up to 2048 x 1536 per display

• DVI resolution up to 1600 x 1200 per display

Note: A single USB keyboard/mouse is supported by the IA2 (base I/O) blade.

The 2D video interface card can be used for all basic interaction with your Altix system. You should note that it does not provide an interface to the L1/L2 controller and L1/L2 administrative commands cannot be issued through the 2D video interface.

58 007-4857-002

Installing Optional Components

DVDROM

PCI-X 0

PCI-X 1

HDD0 HDD1

2D graphics car

ETH0 ETH1

P0 PF N0 N1

F1 F2 F3F4

A1 A2 A3 A4

SAS 4X USB0 USB1 USB2 USB3

Figure 2-4 2D Graphics Card Location in IA2 (base I/O) Blade

Installing Optional Components

Besides adding a system console, you can add or replace the following hardware items on your Altix 450 series server:

• Peripheral component interface (PCI) cards into your system I/O blades.

• Disk drives in your IA/IA2 blade (base I/O).

The sections that follow discuss these activities in more detail.

007-4857-002 59

2: Operation Procedures

I/O Blade Overview

W arning: You can add or replace only the items listed in this section. For your safety and

for the protection of your server system, contact your SGI system support engineer (SSE) to install any hardware items not listed in this section.

Warning: Before installing, operating, or servicing any part of this product, read the

“Safety Information” on page 165.

System I/O is primarily accomplished through standard and optional blades in the SGI Altix 450 IRUs. The PCI, PCI-X and PCIe based I/O sub-systems, are industry standard for connecting peripherals, storage and graphics to a processor blade. These are the primary configurable I/O system interfaces for the Altix 450 series systems:

• The IA/IA2 blade (base I/O) used in the Altix 450 system includes two half-height PCI/PCI-X slots. The two option cards install adjacent to the system disk and DVD drives that are also resident in the IA blade. The IA2 blade differs from the original IA base I/O in that it supports RAID 1 disk mirroring and DVD-R/W capabilities.

• The optional three-slot PCI/PCI-X double-wide blade holds three sled-mounted PCI/PCI-X cards for easy insertion and extraction from the system. It is only used in A450 systems that have more than one IRU.

• The optional single-wide two-slot PCI-X blade supports two PCI or PCI-X option cards.

• The optional single-wide PCI express (PCIe) blade supports two optional PCI express cards.

• The optional double-wide PCI express (PCIe) blade is used only in Altix 450 systems with two or more IRUs. It houses two PCIe slots and two PCI-X slots. Note that use of two PCIe (3D) graphics cards in this blade precludes using the two PCI-X slots due to space limitations.

Not all blades may be available with your system configuration. Check with your SGI sales or service representative for availability . See Chapter 6, “Maintenance and Upgrade Procedures” for detailed instructions on installing or removing PCI cards and blades.

60 007-4857-002

Adding or Removing Disk Drives

The IA/IA2 blade (base I/O blade) within the IRU supports the system boot functions and contains one or two low-profile disk drives. See “Installing or Replacing a Disk Drive in the IA Blade” on page 138 for detailed instructions on installing or removing disk drives.

For information on removing or replacing disk drives in optional mass storage enclosures, see the hardware document supplied with the storage option.

Reconfigure the System Disk Drive(s) for RAID 1

System disk drive(s) ordered starting in March 2007 may come configured as either a non-RAID (jbod) or as a RAID 1 (disk mirror) system. Note that while RAID 0 (striping) is supported by the hardware, SGI recommends using RAID 1 to ensure maximum reliability an d data retention.

The type of disk configuration is ordered by the customer and configured at the SGI factory. If a requirement exits to reconfigure the system disk(s), use the following information to make the changes needed. The lsiutil configuration software is the only supported methodology for modifying the disks at the time this document was published. Use of other software interfaces has not been verified and cannot be supported.

Installing Optional Components

Important: The example process that follows presumes there are two disks identical in size and speed installed in your Altix system IA2 (base I/O) blade. Also, note that this example of re-imaging a disk is done via network access.

The lsiutil file (efi version) resides at /boot/efi

Launch the configuration utility with the command:

fs0:\> lsiutil64.efi

You should see an interactive interface similar to the following:

LSI Logic MPT Configuration Utility, Version 1.xx, November 7, 200X 1 MPT Port found Port Name Chip Vendor/Type/Rev MPT Rev Firmware Rev IOC

1. 01/00/01/0 LSI Logic SAS1068 B0 105 01100000 0

The SAS1068 is the chip that controls the device that the lsiutil utility was loaded from.

007-4857-002 61

Select a device: [1-1 or 0 to quit] 1

1. Identify firmware, BIOS, and/or FCode

2. Download firmware (update the FLASH)

4. Download/erase BIOS and/or FCode (update the FLASH)

8. Scan for devices

10. Change IOC settings (interrupt coalescing)

13. Change SAS IO Unit settings

16. Display attached devices

18. Change WWID

20. Diagnostics

21. RAID actions

22. Reset bus

23. Reset target

24. Clear ACA

39. Force firmware download boot

45. Concatenate SAS firmware and NVDATA files

60. Show non-default settings

61. Restore default settings

97. Reset SAS phy

98. Reset SAS link

99. Reset port

e Enable expert mode in menus p Enable paged mode in menus w Enable logging

Main menu, select an option: [1-99 or e/p/w or 0 to quit] 8

Important: After this command is entered, the load device cannot be accessed again to open files until lsiutil has been exited and restarted.

SAS1068's links are 3.0 G, 3.0 G, down, down, down, down, down, down

B___T___L Type Vendor Product Rev SASAddress PhyNum 0 0 0 Disk SGI ST3146854SS X422 5000c5000002cb45 0 0 1 0 Disk SGI ST3146854SS X422 5000c500000121cd 1

Main menu, select an option: [1-99 or e/p/w or 0 to quit] 0

Port Name Chip Vendor/Type/Rev MPT Rev Firmware Rev IOC

1. 01/00/01/0 LSI Logic SAS1068 B0 105 01100000 0

Select a device: [1-1 or 0 to quit] 1

Installing Optional Components

1. Identify firmware, BIOS, and/or FCode

2. Download firmware (update the FLASH)

4. Download/erase BIOS and/or FCode (update the FLASH)

8. Scan for devices

10. Change IOC settings (interrupt coalescing)

13. Change SAS IO Unit settings

16. Display attached devices

18. Change WWID

20. Diagnostics

21. RAID actions

22. Reset bus

23. Reset target

24. Clear ACA

39. Force firmware download boot

45. Concatenate SAS firmware and NVDATA files

60. Show non-default settings

61. Restore default settings

97. Reset SAS phy

98. Reset SAS link

99. Reset port e Enable expert mode in menus p Enable paged mode in menus w Enable logging

Main menu, select an option: [1-99 or e/p/w or 0 to quit] 21

1. Show volumes

2. Show physical disks

3. Get volume state

23. Replace physical disk

30. Create volume

31. Delete volume

32. Change volume settings

50. Create hot spare

99. Reset port e Enable expert mode in menus p Enable paged mode in menus w Enable logging

RAID actions menu, select an option: [1-99 or e/p/w or 0 to quit] 1

0 volumes are active, 0 physical disks are active

RAID actions menu, select an option: [1-99 or e/p/w or 0 to quit] 30

007-4857-002 63

2: Operation Procedures

B___T___L Type Vendor Product Rev Disk Blocks Disk MB

1. 0 0 0 Disk SGI ST3146854SS X422 286749488 140014

2. 0 1 0 Disk SGI ST3146854SS X422 286749488 140014

To create a volume, select two or more of the available targets

Select a target: [1-2 or RETURN to quit] 1 Select a target: [1-2 or RETURN to quit] 2

2 physical disks were created

Select volume type: [0=Mirroring, 1=Striping, default is 0] Select volume size: [1 to 139898 MB, default is 139898] Enable write caching: [Yes or No, default is No] Zero the first and last blocks of the volume? [Yes or No, default is No]

Volume was created

RAID actions menu, select an option: [1-99 or e/p/w or 0 to quit] 1

1 volume is active, 2 physical disks are active

Volume 0 is Bus 0 Target 0, Type IM (Integrated Mirroring) Volume State: degraded, enabled , resync in progress Volume Settings: write caching disabled , auto configure Volume Size 139898 MB, Stripe Size 0 KB, 2 Members Primary is PhysDisk 0 (Bus 0 Target 56) Secondary is PhysDisk 1 (Bus 0 Target 1)

RAID actions menu, select an option: [1-99 or e/p/w or 0 to quit] 99

Resetting port...

RAID actions menu, select an option: [1-99 or e/p/w or 0 to quit] 0

Main menu, select an option: [1-99 or e/p/w or 0 to quit] 0

Port Name Chip Vendor/Type/Rev MPT Rev Firmware Rev IOC

1. 01/00/01/0 LSI Logic SAS1068 B0 105 01100000 0

Select a device: [1-1 or 0 to quit] 0 Sas(Pun0,Lun0) LSILOGICLogical Volume 3000

64 007-4857-002

Installing Optional Components

fs0:\> reset . . . EFI Boot Manager ver 1.10 [14.62]

Partition 0: Enabled Disabled CBlades 2 Nodes 3 0 RBlades 0 CPUs 4 0 IOBlades 1 Mem(GB) 6 0

Please select a boot option

EFI Shell >>> netboot Boot option maintenance menu

Loading: netboot Running LoadFile()

CLIENT MAC ADDR: 08 00 69 14 E3 0C CLIENT IP: 137.38.82.91 MASK: 255.255.255.0 DHCP IP: 137.38.228.4 GATEWAY IP: 137.38.82.252 137.38.82.254

TSize.Running LoadFile()

Starting: netboot ELILO . . . . Quietly populating /dev/sda2 and /dev/sda1 with ia64-propack51-sles10-Jan19

# reboot....

Note: The initial resynchronization of the mirrored disk (depending on the size of the disk) may take up to 45 minutes or longer.

RAID actions menu, select an option: [1-99 or e/p/w or 0 to quit] 1

007-4857-002 65

2: Operation Procedures

1 volume is active, 2 physical disks are active

Volume 0 is Bus 0 Target 0, Type IM (Integrated Mirroring) Volume State: optimal, enabled Volume Settings: write caching disabled, auto configure Volume draws from Hot Spare Pools: 0 Volume Size 139898 MB, Stripe Size 0 KB, 2 Members Primary is PhysDisk 1 (Bus 0 Target 56) Secondary is PhysDisk 0 (Bus 0 Target 1)

For information on the two RAID 1 disks, use the lsscsi command as shown in the following example:

# lsscsi

[1:0:0:0] disk SGI ST3146854SS X422 - /dev/sg2 [1:0:1:0] disk SGI ST3146854SS X422 - /dev/sg3 [1:1:0:0] disk LSILOGIC Logical Volume 3000 /dev/sdc /dev/sg4

This is an example of what you'll see on SLES10 or RHEL5 Linux operating systems. Disk [1:0:0:0] and [1:0:1:0] are the individual drives in the RAID 1, but you cannot access them through the disk driver. You access [1:1:0:0], which is /dev/sdc in this case. Caution must be exercised when entering Linux SCSI generic (sg) commands via the sg devices.

66 007-4857-002

Chapter 3

3. System Overview

This chapter provides an overview of the physical and architectural aspects of your SGI Altix 450 system. The major components of the Altix 450 series systems are described and illustrated.

The Altix 450 series is a family of multiprocessor distributed shared memory (DSM) computer systems that initially scale from 2 to 76 Intel 64-bit processor cores as a cache-coherent single system image (SSI). Contact your SGI sales or service representative for the most current information on this topic.

In a DSM system, each processor board contains memory that it shares with the other processors in the system. Because the DSM system is modular, it combines the advantages of low entry-level cost with global scalability in processors, memory, and I/O. You can install and operate the Altix 450 series system in your lab or server room. Each 20U SGI rack holds from one to four 5U high enclosures that support up to five compute/memory and I/O sub modules known as “blades.” These blades are single printed circuit boards (PCBs) with ASICS, processors, memory components and I/O chipsets mounted on a mechanical carrier. The blades slide directly into and out of the Altix 450 IRU enclosures.

This chapter consists of the following sections:

• “System Models” on page 68

• “System Architecture” on page 70

• “System Features” on page 72

• “System Components” on page 76

Figure 3-1 shows the front view of a 20U rack used to house the Altix 450 system.

007-4857-002 67

3: System Overview

System Models

Figure 3-1 SGI Altix 450 System (Short Rack)

The basic enclosure within the Altix 450 system is the 5U high (8.68 inch or 22 cm) “individual rack unit” (IRU). The IRU enclosure houses a maximum of four single-wide blades and one double-wide blade. Each IRU comes with two built-in high-speed routers. The routers connect to the installed blades via a backplane. Each router has two ports that are brought out to external NUMAlink 4 connectors. The 20U or 42U rack for this server houses all IRU enclosures, option modules, and other components; up to a 76-processor core configuration (38 processor sockets) in a single rack. The Altix 450 server system needs a minimum of one IA/IA2 blade (base I/O).

The rack system requires a minimum of one 20U-high rack with one single-phase power distribution unit (PDU) installed in the rack. Each single-phase PDU has 5 outlets (to support two IRUs).

Figure 3 -2 shows an example configuration of a 42U Altix 450 system “tall rack.”

68 007-4857-002

System Models

Figure 3-2 SGI Altix 450 Server System (Tall Rack)

You can also add additional PCI expansion blades or RAID and non-RAID disk storage to your server system.

Figure 3-3 shows an individual blade, IRU and Rack.

007-4857-002 69

3: System Overview

Blade

Individual Rack Unit (IRU)

(Contains 5 blades)

Blade slot 0

Figure 3-3 Blade, IRU and Rack Components

Blade slot 1

Blade slot 2

Blade slot 3

Blade slot 4

Rack

(Contains 4 IRUs)

inside

System Architecture

The Altix 450 computer system is based on a distributed shared memory (DSM) architecture. The system uses a global-address-space, cache-coherent multiprocessor that scales up to 76 Intel 64-bit processor cores in a single rack. Because it is modular, the DSM combines the advantages of lower entry cost with the ability to scale processors, memory, and I/O independently.

The system architecture for the Altix 450 system is a fourth-generation NUMAflex DSM architecture known as NUMAlink 4. In the NUMAlink 4 architecture, all processors and memory are tied together into a single logical system with special crossbar switches (routers). This combination of processors, memory, and crossbar switches constitute the interconnect fabric called NUMAlink. There are two internal router switches on each 5U IRU enclosure.

The basic expansion building block for the NUMAlink interconnect is the processor node; each processor node consists of a Super-Hub (SHub) ASIC and one or two 64-bit processors with three levels of on-chip secondary caches. The Intel 64-bit processors are connected to the SHub ASIC via a single high-speed front side bus.

The SHub ASIC is the heart of the processor and memory node blade technology. This specialized ASIC acts as a crossbar between the processors, local SDRAM memory, and the network

70 007-4857-002

System Architecture

interface. The SHub ASIC memory interface enables any processor in the system to access the memory of all processors in the system.

Another component of the NUMAlink 4 architecture is the router ASIC. The router ASIC is a custom designed 8-port crossbar ASIC. Using the router ASICs with a highly specialized backplane and NUMAlink 4 cables provides a high-bandwidth, extremely low-latency interconnect between all processor, I/O, and other option blades within the system.

Figure 3-4 shows a functional block diagram of the Altix 450 series IRU components.

Blade Slot 0

Left router board

(Slab 0)

NUMAlink

System control

Figure 3-4 Functional Block Diagram of the Individual Rack Unit (IRU)

Blade Slot 1

DEF

rtr

Power Supply Power Supply

Blade Slot 2

Blade Slot 3

Blade Slot 4

rtr

Right router board

BCD G

(Slab 1)

NUMAlink

DC power bus

007-4857-002 71

3: System Overview

System Features

The main features of the Altix 450 series server systems are introduced in the following sections:

• “Modularity and Scalability” on page 72

• “Distributed Shared Memory (DSM)” on page 72

• “Distributed Shared I/O” on page 73

• “Reliability, Availability, and Serviceability (RAS)” on page 74

Modularity and Scalability

The Altix 450 series systems are modular systems. The components are primarily housed in building blocks referred to as individual rack units (IRUs). Additional optional mass storage may be added to the rack along with additional IRUs. You can add different types of blade options to a system IRU to achieve the desired system configuration. You can easily configure systems around processing capability, I/O capability, memory size, or storage capacity. You place individual blades that create the basic functionality (compute/memory, I/O, and power) into IRUs. The air-cooled IRU enclosure system has redundant, hot-swap fans and redundant, hot-swap power supplies at the IRU level.

Distributed Shared Memory (DSM)

In the Altix 450 system, memory is physically distributed both within and among the IRU enclosures (compute/memory/I/O blades); however, it is accessible to and shared by all NUMAlinked devices within the single-system image. This is to say that all NUMAlinked components sharing a single Linux operating system operate and share the memory “fabric” of the system.

Note the following sub-types of memory within a system:

• If a processor accesses memory that is connected to the same SHub ASIC on a compute node blade, the memory is referred to as the node’s local memory.

• If processors access memory located in other blade nodes within the IRU, (or other NUMAlinked IRUs) the memory is referred to as remote memory.

• The total memory within the NUMAlinked system is referred to as global memory.

72 007-4857-002

Distributed Shared I/O

ccNUMA Architecture

Cache Coherency

System Features

Memory latency is the amount of time required for a processor to retrieve data from me mory. Memory latency is lowest when a processor accesses local memory.

Like DSM, I/O devices are distributed among the blade nodes within the IRUs (each base I/O blade node has two NUMAlink ports) and are accessible by all compute nodes within the SSI through the NUMAlink interconnect fabric.

As the name implies, the cache-coherent non-uniform memory access (ccNUMA) architecture has two parts, cache coherency and nonuniform memory access, which are discussed in the sections that follow.

The Altix 450 systems use caches to reduce memory latency. Although data exists in local or remote memory, copies of the data can exist in various processor caches throughout the system. Cache coherency keeps the cached copies consistent.

T o keep the copies consistent, the ccNUMA architecture uses directory-based coherence protocol. In directory-based coherence protocol, each block of memory (128 bytes) has an entry in a table that is referred to as a directory. Like the blocks of memory that they represent, the directories are distributed among the compute/memory blade nodes. A block of memory is also referred to as a cache line.

Each directory entry indicates the state of the memory block that it represents. For example, when the block is not cached, it is in an unowned state. When only one processor has a copy of the memory block, it is in an exclusive state. And when more than one processor has a copy of the block, it is in a shared state; a bit vector indicates which caches contain a copy.

When a processor modifies a block of data, the processors that have the same block of data in their caches must be notified of the modification. The Altix 450 server series use an invalidation method to maintain cache coherence. The invalidation method purges all unmodified copies of the block of data, and the processor that wants to modify the block receives exclusive ownership of the block.

007-4857-002 73

3: System Overview

Non-uniform Memory Access (NUMA)

In DSM systems, memory is physically located at various distances from the processors. As a result, memory access times (latencies) are different or “non-uniform.” For example, it takes less time for a processor blade to reference its locally installed memory than to reference remote memory.

Reliability, Availability, and Serviceability (RAS)

The Altix 450 server series components have the following features to increase the reliability, availability, and serviceability (RAS) of the systems.

• Power and cooling:

– IRU power supplies are redundant and can be hot-swapped under most circumstances.

Note that this might not be po ssib le in a “f ully loaded” system. If all the blade positions are filled, be sure to consult with a service technician before removing a power supply

while the system is running. – IRUs have overcurrent protection at the blade and power supply level. – Fans are redundant and can be hot-swapped. – Fans run at multiple speeds in the IRUs. Speed increases automatically when

temperature increases or when a single fan fails.

• System monitoring:

– System controllers monitor the internal power and temperature of the IRUs, and can

automatically shut down an enclosure to prevent overheating. – Memory, L2 cache, L3 cache, and all external bus transfers are protected by single-bit

error correction and double-bit error detection (SECDED). – The NUMAlink interconnect network is protected by cyclic redundancy check (CRC). – The L1 primary cache is protected by parity. – Each IRU and each blade/node installed has failure LEDs that indicate the failed part;

LEDs are readable at the front of the IRU or via the system controllers. – Systems support the optional Embedded Support Partner (ESP), a tool that monitors the

system; when a condition occurs that may cause a failure, ESP notifies the appropriate

SGI personnel. – Systems support remote console and maintenance activities.

74 007-4857-002

System Features

• Power-on and boot:

– Automatic testing occurs after you power on the system. (These power-on self-tests or

POSTs are also referred to as power-on diagnostics or PODs). – Processors and memory are automatically de-allocated when a self-test failure occurs. – Boot times are minimized.

• Further RAS features:

– Optional RAID 1 in base I/O (IA2 blade), check with your SGI sales or service

representative for availability. Note that RAID 0 (striping), while supported by the IA2

hardware, is not recommended as a RAS option. – Systems have a local field-replaceable unit (FRU) analyzer. – All system faults are logged in files. – Memory can be scrubbed using error checking code (ECC) when a single-bit error

occurs.

007-4857-002 75

3: System Overview

System Components

The Altix 450 series system features the following major components:

• 20U rack. The “short” rack is a custom rack used with the Altix 450 system. It holds up to 4

• 42U rack. The “tall” rack is a custom rack used for both the compute and I/O rack in the

• Individual Rack Unit (IRU). This enclosure contains the compute/memory blades, IA

• Compute/Memory blade. Holds up to two IA-64 processor sockets and 4, 8 or 12 memory

• Memory-only blade. This blade acts as a memory expansion node with no processor

• Single-wide PCI-X expansion blade. This two-slo t PCI/PCI-X opt ion blade su pplies an

IRUs or a combination of IRUs and option modules (such as mass storage).

Altix 450 system. Up to 8 IRUs can be installed in each rack. There is also a 2U space reserved at the top for an option module.

blade (base I/O), standard routers and optional I/O blades for the Altix 450 series systems. The enclosure is 5U high.

DIMMs.

compute circuitry included on the blade. This blade holds 4, 8, or 12 memory DIMMs.

individual PCI bus for each option card.

• Double-wide PCI-X expansion blade. Supports three PCI/PCI-X 133 MHz 64-bit option

cards. This three-slot blade features card carriers that allow you to slide PCI/PCI-X boards directly into and out of the unit.

• Single-wide PCIe expansion blade. The single wide PCI blade supports one or two PCI

Express option cards. Note that when used with a 3D PCIe graphics card, only one can be inserted due to space constraints.

• Double-wide PCIe/PCI-X expansion blade. This blade supports two PCI Express option

cards and two PCI/PCI-X option cards. Note that when this blade is used with two 3D PCIe graphics cards the PCI-X slots cannot be used due to space constraints.

• IA/IA2 blade (Base I/O blade). Double-wide I/O blade that supports all base system I/O

functions including one or two disk drives, a DVD drive, two low-profile PCI-X card slots, two ethernet ports, one SAS/SATAII port, and four USB ports. Optional RAID 1 functionality and DVD-R/W is available with the IA2 version of the base I/O blade only.

Figure 3-5 on page 77 shows the Altix 450 IRU system components.

76 007-4857-002

System Components

L1 controller

panel

ort

Bay (Unit) Numbering

PCI-X 1 PCI-X 0

DVDROM

HDD0 HDD1

ETH0 ETH1

P0PF N0N1

F1F2 F3 F4

A1A2 A3A4

SAS 4X USB0 USB1 USB2 USB3

inside

Power

Router

board

IA blade

Compute/memory blades

System control

board

supplies

Router

board

Figure 3-5 Altix 450 IRU System Components Example

Bays in the racks are numbered using standard units. A standard unit (SU) or unit (U) is equal to

1.75 inches (4.445 cm). Because IRUs occupy multiple standard units, IRU locations within a rack are identified by the bottom unit (U) in which the IRU resides. For example, in a 42U rack, an IRU positioned in U01 through U05 is identified as U01.

Po G

Po H

Rack Numbering

Each rack is numbered with a three-digit number sequentially beginning with 001. A rack contains

007-4857-002 77

IRU enclosures, optional mass storage enclosures, and potentially other options. In a single compute rack system, the rack number is always 001.

3: System Overview

Optional System Components

Availability of optional components for the SGI 450 systems may vary based on new prod uct introductions or end-of-life components. Some options are listed in this manual, others may be introduced after this document goes to production status. Check with your SGI sales or support representative for the most current information on available product options not discussed in this manual.

78 007-4857-002

Silicon Graphics SGI Altix 450 User Manual

Specifications and Main Features

Frequently Asked Questions

User Manual