Sun Microsystems CP3260 User Manual

Sun Netra™CP3260Blade Server

User’s Guide

Sun Microsystems, Inc. www.sun.com

Part No. 820-0457-11 April 2009, Revision 01

Submit comments about this document at: http://www.sun.com/hwdocs/feedback

This distributionmay include materials developed bythird parties.

Parts ofthe product may be derivedfrom BerkeleyBSD systems, licensed from the University ofCalifornia. UNIX is a registered trademark in the U.S.and in other countries, exclusivelylicensed through X/Open Company, Ltd.

Sun, SunMicrosystems, the Sun logo, Netra,Sun Ray, theNetra logo and the Solarislogo are trademarks or registered trademarks of Sun Microsystems, Inc.,and its subsidiaries, in theU.S. and other countries.

All SPARC trademarks are used under license andare trademarksor registeredtrademarks of SPARC International, Inc. inthe U.S. and other countries. Productsbearing SPARC trademarks are based upon architecture developed by SunMicrosystems, Inc.

Use ofany spare or replacement CPUs is limitedto repair or one-for-one replacement of CPUs in productsexported in compliance with U.S. export laws.Use of CPUs as productupgrades unless authorized by theU.S. Government is strictly prohibited.

DOCUMENTATION IS PROVIDED "AS IS" AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY,FITNESS FOR APARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.

Cette distributionpeut comprendredes composants développés par destierces parties.

Des partiesde ce produit pourront être dérivées des systèmes Berkeley BSD licenciés par l’Université deCalifornie. UNIX est une marque déposée auxEtats-Unis et dans d’autres payset licenciée exclusivement par X/OpenCompany, Ltd.

Sun, SunMicrosystems, le logo Sun, Netra,Sun Ray, lelogo Netra et le logoSolaris sont des marques de fabrique oudes marques déposées de Sun Microsystems,Inc. ou ses ﬁliales, auxEtats-Unis et dans d’autres pays.

Toutes les marquesSPARCsont utilisées sous licence etsont des marques de fabrique ou desmarques déposées de SPARC International, Inc. aux Etats-Uniset dans d’autres pays. Lesproduits portantles marques SPARC sont basés sur une architecture développée parSun Microsystems, Inc.

see aboveL’utilisation de pieces detacheesou d’unites centrales de remplacement est limiteeaux reparations ou a l’echangestandard d’unites centrales pourles produits exportes, conformement ala legislation americaine en matiere d’exportation. Saufautorisation par les autorites des Etats-Unis, l’utilisationd’unites centrales pour proceder ades mises a jour deproduits estrigoureusement interdite.

LA DOCUMENTATION EST FOURNIE "EN L’ETAT" ET TOUTES AUTRES CONDITIONS, DECLARATIONS ET GARANTIES EXPRESSES OU TACITES SONT FORMELLEMENT EXCLUES, DANSLA MESURE AUTORISEE PAR LA LOI APPLICABLE, YCOMPRIS NOTAMMENT TOUTE GARANTIE IMPLICITE RELATIVE A LA QUALITE MARCHANDE, A L’APTITUDE A UNE UTILISATION PARTICULIERE OU A L’ABSENCE DE CONTREFACON.

Please

Recycle

Preface xiii

1. Introduction 1–1

1.1 Overview 1–1

1.2 Features 1–2

1.3 System Configurations 1–8

1.4 Advanced Rear Transition Modules 1–11

1.5 Hot-Swap Support 1–14

1.6 System Requirements and Options 1–14

1.6.1 Hardware Requirements and Options 1–14

1.6.2 Software Requirements 1–15

1.7 Technical Support and Warranty 1–15

1.7.1 Part Number, Serial Number, and Revision Number Identification 1–16

2. Hardware Installation 2–1

2.1 Equipment and Operator Safety 2–1

2.2 Materials and Tools Required 2–3

2.3 Preparing for the Installation 2–3

2.3.1 Checking Power, Thermal, Environmental, and Space Requirements 2–3

iii

2.4 Installation Procedure Summary 2–4

2.5 Configuring On-Board Hardware 2–5

2.5.1 Adding or Replacing FB-DIMM Memory Modules 2–5

2.5.1.1 To Remove FB-DIMM Memory Modules 2–10

2.5.1.2 To Install FB-DIMM Memory Modules 2–11

2.5.2 Adding or Replacing Compact Flash Card 2–13

2.5.3 Adding or Replacing TOD Clock Battery 2–14

2.6 Installing an Advanced Rear Transition Module (Optional) 2–18

2.7 Installing the Netra CP3260 Blade Server 2–21

2.8 Connecting External I/O Cables 2–23

2.9 Hot-Swapping the Netra CP3260 Blade Server 2–25

3. Software Installation 3–1

3.1 Operating Systems and Patches 3–1

3.2 Configuring Payload OS NIU Driver for Multiplexing to Zones 2 and 3 3– 2

3.3 Installing as a Diskless Clients 3–4

3.3.1 Creating a Boot Server for Diskless Clients 3–4

3.3.2 Adding a Diskless Client 3–5

3.3.3 Determining Local Network IP Addresses and Host Names 3–7

3.4 Firmware Updates 3–8

3.5 Configuring for 1-GbE or 10-GbE Switches 3–8

3.6 Downloading and Installing SunVTS Software 3–9

3.7 Formatting the Optional Compact Flash Card 3–10

4. Firmware and Blade Server Management 4–1

4.1 System Firmware 4–2

4.2 Power-On Self-Test Diagnostics 4–3

4.2.1 POST Test Coverage 4–3

4.2.2 POST Diagnostic and Error Message Format 4–4

iv Netra CP3260 Blade Server User’s Guide • April 2009

4.3 OpenBoot Firmware 4–4

4.3.1 Getting to the ok Prompt 4–5

4.3.2 Auto-Boot Options 4–6

4.3.3 OpenBoot Commands 4–6

4.3.3.1 probe-scsi and probe-scsi-all Commands 4–7

4.3.3.2 probe-ide Command 4–8

4.3.3.3 show-devs Command 4–8

4.3.3.4 Using watch-net and watch-net-all Commands to Check the Network 4–11

4.3.4 OpenBoot Configuration Variables 4–12

4.3.4.1 Viewing and Setting OpenBoot Configuration Variables 4–14

4.4 Error Handling Summary 4–15

4.5 Automatic System Recovery 4–16

4.5.1 Enabling and Disabling Automatic System Recovery 4–17

4.5.1.1 To Enable Automatic System Recovery 4–17

4.5.1.2 To Disable Automatic System Recovery 4–17

4.6 Network Device Aliases 4–18

4.7 Retrieving Device Information 4–19

4.8 Multiplexing to Zones 2 and 3 4–27

5. Hardware Functional Descriptions 5–1

5.1 Hardware Architecture 5–1

5.1.1 UltraSPARC T2 Processor and Memory 5–2

5.1.2 I/O 5–3

5.1.3 Base and Fabric Interfaces 5–3

5.1.4 Additional I/O 5–3

5.1.5 Power PC 5–3

5.1.6 ARTM Support 5–4

5.2 Hardware Modules 5–4

Contents v

5.2.1 UltraSPARC T2 Processor 5–4

5.2.1.1 Electronic Fuse 5–7

5.2.1.2 Cores 5–8

5.2.1.3 L2 Cache 5–8

5.2.1.4 Memory Controller 5–8

5.2.1.5 I/O Interface 5–8

5.2.2 Memory Subsystem 5–9

5.2.2.1 Memory Capacity 5–9

5.2.2.2 Memory Speed 5–9

5.2.3 I/O Subsystem 5–9

5.2.3.1 PCI Express Switch 5–10

5.2.3.2 Base Interface 5–10

5.2.3.3 Fabric Interface 5–10

5.2.3.4 Common ARTM 5–10

5.2.4 Other ARTM Interfaces 5–11

5.2.4.1 Serial Ports 5–11

5.2.4.2 Ethernet Management Port 5–11

5.2.5 Front Panel I/O 5–11

5.2.5.1 Ethernet Management Port 5–11

5.2.5.2 Serial Port 5–11

5.2.5.3 Dual USB Ports 5–12

5.2.6 Compact Flash Socket 5–12

5.2.7 Service Processor MPC885 5–12

5.2.7.1 Field-Programmable Gate Array 5–12

5.2.8 Intelligent Platform Management Controller 5–13

5.2.8.1 Intelligent Platform Management Bus 5–14

5.2.8.2 Interface to the PPC 5–14

5.2.8.3 IPMB-L Interface 5–14

vi Netra CP3260 Blade Server User’s Guide • April 2009

5.2.8.4 ATCA Hot-Swap Latch 5–14

5.2.8.5 LEDs 5–15

5.2.8.6 Power Control 5–15

5.2.8.7 System Monitor (ADM1026) and Thresholds 5–15

5.2.8.8 FRUID PROMs 5–17

5.2.9 I/O Subsystem Resets 5–17

5.2.10 ATCA Power Module (−48V to 12V) 5–18

5.2.11 TOD Clock Battery 5–18

A. Physical Characteristics A–1

A.1 Form Factor A–1

A.2 Layout A–1

A.3 Front Panel A–3

A.3.1 Visual Indicators A–3

A.3.2 Ports A–3

A.4 Connectors and Pinout A–3

A.4.1 Front Panel Connectors A–3

A.4.1.1 Ethernet Port A–4

A.4.1.2 Dual USB Port A–4

A.4.1.3 Serial Port A–5

A.4.2 Compact Flash Connector A–6

A.4.3 Midplane Power Connector (Zone 1) A–6

A.4.4 Data Transport Connector (Zone 2) A–8

A.4.5 Sun Netra ARTM Connectors (Zone 3) A–10

A.4.6 TOD Clock Battery Holder A–13

B. Sun OEM IPMI Commands B–1

B.1 Get Version Command B–2

B.2 Get RTM Status Command B–3

Contents vii

B.3 Solaris OS Graceful Shutdown Commands B–4

B.4 Send Sensor State Command B–5

Index Index–1

viii Netra CP3260 Blade Server User’s Guide • April 2009

Figures

FIGURE 1-1 Netra CP3260 Blade Server Front Panel 1–4

FIGURE 1-2 Netra CP3260 Blade Server (Top View) 1–5

FIGURE 1-3 Netra CP3260 Blade Server in an ATCA Shelf Enclosure 1–9

FIGURE 1-4 Netra CP3260 Blade Server, Midplane, and Netra CP32x0 ARTM 1–12

FIGURE 1-5 Netra CP3260 Blade Server Barcode Labeling 1–17

FIGURE 2-1 FB-DIMM Memory Locations 2–6

FIGURE 2-2 Removing an FB-DIMM Memory Module 2–11

FIGURE 2-3 Installing a FB-DIMM Memory Module 2–12

FIGURE 2-4 Compact Flash Card Location 2–14

FIGURE 2-5 TOD Battery Location 2–16

FIGURE 2-6 Installing a Netra CP32X0 ARTM 2–19

FIGURE 2-7 Installing Blade Server Into Chassis Slot 2–22

FIGURE 2-8 Netra CP3260 Blade Server Latches and Locking Screws 2–23

FIGURE 2-9 Hot-Swap Latch and Hot-Swap LED 2–26

FIGURE 5-1 Netra CP3260 Blade Server Block Diagram 5–2

FIGURE 5-2 UltraSPARC T2 Multicore Processor Block Diagram 5–5

FIGURE A-1 Netra CP3260 Blade Server Layout A–2

FIGURE A-2 Ethernet RJ-45 Connector A–4

FIGURE A-3 Dual USB Connector A–5

FIGURE A-4 Front Panel Serial Port Diagram A–6

FIGURE A-5 Power Distribution Connector (Zone 1) P10 A–7

FIGURE A-6 Zone 2 Connectors A–9

FIGURE A-7 Zone 3 Signal Connectors A–10

FIGURE A-8 Zone 3 Power Connector A–12

FIGURE A-9 TOD Battery Location A–14

x Netra CP3260 Blade Server User’s Guide • April 2009

Tables

TABLE 1-1 Netra CP3260 Blade Server Feature Summary 1–2

TABLE 1-2 I/O Configurations 1–13

TABLE 2-1 Ethernet Device Names 2–24

TABLE 3-1 Local Network Information 3–7

TABLE 4-1 Ways of Accessing the ok Prompt 4–5

TABLE 4-2 OpenBoot Configuration Variables 4–12

TABLE 4-3 Network Device Aliases 4–18

TABLE 4-4 PICL Frutree Entries and Description for the Netra CP3260 Board 4–19

TABLE 5-1 Voltage Sensor Thresholds 5–16

TABLE 5-2 CPU Temperature Alarms 5–16

TABLE A-1 Ethernet Port Connector Pin Assignments A–4

TABLE A-2 USB Connector Pin Assignments A–5

TABLE A-3 Serial Port RJ-45 Connector Pinouts A–6

TABLE A-4 Power Distribution Connector Pin Assignments A–7

TABLE A-5 Zone 2 J23 Connector Pin Assignments A–9

TABLE A-6 Zone 2 J20 Connector Pin Assignments A–9

TABLE A-7 Zone 3 J31 Connector Pin Assignments A–11

TABLE A-8 Zone 3 J32 Connector Pin Assignments A–11

TABLE A-9 Zone 3 J33 Connector Pin Assignments A–12

TABLE A-10 Zone 3 Power Connector Pin Assignments A–13

TABLE B-1 Get Version Command Data Bytes B–2

TABLE B-2 Get RTM Status Command Data Bytes B–3

TABLE 1 Solaris OS Graceful Shutdown Parameters in /etc/fsmd.conf B–4

TABLE B-3 Send Sensor State Command Data Bytes B–5

xii Netra CP3260 Blade Server User’s Guide • April 2009

Preface

The Sun Netra CP3260 Blade Server User ’s Guide provides information about features, installation, configuration, functional hardware components, and physical properties of this blade server. The Sun Netra CP3260 Blade Server User’s Guide is written for system integration engineers, field applications and service engineers, and others involved in the integration of these blade servers into systems.

How This Document Is Organized

Chapter 1 provides an overview of the Sun Netra™ CP3260 blade server.

Chapter 2 provides instructions on hardware installation.

Chapter 3 provides instructions on the software configuration.

Chapter 4 provides information about the Netra CP3260 firmware.

Chapter 5 provides hardware and functional descriptions of the Netra CP3260 blade

server.

Appendix A provides information about the physical characteristics of the Netra

CP3260 blade server.

Appendix B describes the Sun-specific OEM-defined Intelligent Platform

Management Interface (IPMI) commands.

xiii

Using UNIX Commands

This document might not contain information on basic UNIX®commands and procedures such as shutting down the system, booting the system, and configuring devices. Refer to the following for this information:

■ Software documentation that you received with your system

■ Solaris™ Operating System documentation, which is at:

http://docs.sun.com/app/docs/prod/solaris

Shell Prompts

Shell Prompt

C shell machine-name%

C shell superuser machine-name#

Bourne shell and Korn shell $

Bourne shell and Korn shell superuser #

xiv Sun Netra CP3260 Blade Server User’s Guide • April 2009

Typographic Conventions

Typeface Meaning Examples

AaBbCc123 The names of commands, files,

and directories; on-screen computer output

AaBbCc123 What you type, when contrasted

with on-screen computer output

AaBbCc123 Book titles, new words or terms,

words to be emphasized. Replace command-line variables with real names or values.

Edit your.login file.

Use ls -a to list all files.

% You have mail.

Password:

Read Chapter 6 in the User’s Guide.

These are called class options.

Yo u must be superuser to do this.

To delete a file, type rm filename.

Note – Characters display differently depending on browser settings. If characters

do not display correctly, change the character encoding in your browser to Unicode UTF-8.

Introduction

This chapter contains the following sections:

■ Section 1.1, “Overview” on page 1-1

■ Section 1.2, “Features” on page 1-2

■ Section 1.3, “System Configurations” on page 1-8

■ Section 1.5, “Hot-Swap Support” on page 1-14

■ Section 1.6, “System Requirements and Options” on page 1-14

■ Section 1.7, “Technical Support and Warranty” on page 1-15

1.1 Overview

The Netra CP3260 blade server (FIGURE 1-1 and FIGURE 1-2) is a high-performance single-board computer based on one UltraSPARC

T2 multicore processor and designed for high availability in a switched network computing environment. This blade server is compliant with Advanced Telecommunications Computing Architecture (AdvancedTCA

, ATCA®) specifications (PICMG®3.0 and PICMG 3.1).

The PICMG (PCI Industrial Computer Manufacturers Group) standards committee has developed the new ATCA (or PICMG 3.x) standard to address the issues posed by previous standards based on cPCI and cPSB (PICMG 2.x). The PICMG 3.x specification brought the following changes to the existing PICMG 2.x family of products:

■ Larger board space (8U high compared to 6U for cPCI), which allows more

features and processing power

■ On-board power supplies deriving local power from redundant −48V power from

the midplane (rather than separate power supplies)

■ 6-HP slot width, allowing greater component height

■ Advanced Mezzanine card (AMC) support and options

1-1

■ Elimination of PCI connectivity between the blade servers in the system and

reallocation of connectivity to serial interconnects, eliminating single points of failure

■ Mandatory use of Intelligent Platform Management Interface (IPMI) management

interfaces

■ Flexible user I/O

■ Power and thermal management guidelines enforced by the management

infrastructure

■ Separation of control and data traffic by supporting the Base (PICMG 3.0) and

Fabric (PICMG 3.1) interfaces

The ATCA standard consists of the PICMG 3.0, PICMG 3.1, PICMG 3.2, and PICMG

3.3 specifications. The Netra CP3260 blade server complies with:

■ PICMG 3.0, the base specification that defines the mechanical, power distribution,

system management, data transport, and regulatory guidelines

■ PICMG 3.1, which builds upon the PICMG 3.0 base specification and the IEEE

802.3-2003 standard

1.2 Features

TABLE 1-1 provides a summary of Netra CP3260 features.

TABLE 1-1 Netra CP3260 Blade Server Feature Summary

Feature Description

CPU • One UltraSPARC T2 processor (6 or 8 cores, 8 threads/core)

• CPU core speed:

Memory • Fully-Buffered DDR2-based memory DIMMs (FB-DIMMs)

• Eight FB-DIMM slots (one per channel or two per memory controller)

• 1-Gbyte or 2-Gbyte FB-DIMMs, for a total memory of 16 Gbytes

Service processor

Power requirement

Cooling requirement

1-2 Netra CP3260 Blade Server User’s Guide • April 2009

• Service processor (MPC885) for CPU reset, boot, partition, and fault management architecture (FMA)

• 200 watts maximum

• Dual redundant input 48V to derive on-board power

• 32 - 35 cfm at 55C

1.2 GHz

TABLE 1-1 Netra CP3260 Blade Server Feature Summary

Feature Description

PICMG compliance

• Single-wide ATCA module

• ATCA 8U form factor

• Single-wide ATCA module

• 6-HP slot width

• PICMG 3.0 R1.0

• PICMG 3.1 R1.0

Node board support

Operating system

Internal I/O (connections to ATCA midplane)

Functions as a CPU node board with the Solaris operating system and software packages

Solaris 10 8/07 OS and subsequent compatible versions with supported Netra patches

• Dual 10/100/1000BASE-T Ethernet for Base interfaces

• 10-Gb XAUI or 1-Gb SERDES Ethernet interfaces as Fabric interfaces

• Dual IPMI channel connects to the midplane for communicating with the Shelf Management card

External I/O • One 10/100/1000BASE-T Ethernet maintenance port on front panel

• One asynchronous serial port for console on front panel

• Dual USB 2.0 compliant ports on the front panel

• Rear access available on Netra CP32x0 ARTMs:

– One 10/100/1000BASE-T Ethernet port (RJ-45)

– One asynchronous serial port (RJ-45)

Note - With a Netra CP32x0 ARTM installed, both the front panel and ARTM serial ports are active. Either serial port can be used; but only one of the serial ports can be used at a time.

IPMI system management

Uses IPMI communications with baseboard management controller (BMC); performs advanced system monitoring (ASM) on local board interface (for example, temperature sense, FRU ID, and control)

Hot-swap

• Board hot-swap support.

support

Front panel access I/O

• One 10/100/1000BASE-T Ethernet port (RJ-45)

• Dual USB 2.0 compliant ports

• One serial port (RJ-45)

Rear access interface

Compact Flash support

Building

• Common rear transition module (RTM) interface supporting the Sun Netra CP32X0 ARTMs

• Compact Flash socket to support a Sun 8-Gbyte or 16-Gbyte user flash type I/II memory card

Network Equipment Building Systems (NEBS) Level 3

compliance

Flash update Supported from downloaded file

Chapter 1 Introduction 1-3

FIGURE 1-1 Netra CP3260 Blade Server Front Panel

1-4 Netra CP3260 Blade Server User’s Guide • April 2009

FIGURE 1-2 Netra CP3260 Blade Server (Top View)

Chapter 1 Introduction 1-5

⑤

④

③

⑦

1-6 Netra CP3260 Blade Server User’s Guide • April 2009

②

①

Figure Legend

1 Top latch

2 Out of Service LED (yellow)

3 Ethernet management port (RJ-45)

4 OK LED (green)

5 Dual USB Ports (USB 2.0)

6 Serial (console) port (RJ-45, ttya)

7 Hot-Swap LED (blue)

8 Hot-Swap switch and bottom latch

Figure Legend

1 Zone 1 power connector

2 Zone 2 signal connectors

3 Zone 3 RTM signal connectors

4 Zone 3 ARTM power connector

Chapter 1 Introduction 1-7

Figure Legend

5 Compact Flash connector

6 UltraSPARC T2 processor (under heat sink)

7 FB-DIMMs

1.3 System Configurations

Netra CP3260 blade servers can be installed in an ATCA shelf (or chassis), as shown in

FIGURE 1-3. The blade servers can be deployed in various electrical configurations

to suit each end-user requirement. For example, the blade server can be configured to boot from a network as a diskless client with either a front panel or RTM network connection, or from an optional Compact Flash card. Also, a Netra CP32x0 SAS Storage ARTM, Dual HD can be installed to provide local disk I/O, which can be used optionally as a boot path.

1-8 Netra CP3260 Blade Server User’s Guide • April 2009

FIGURE 1-3 Netra CP3260 Blade Server in an ATCA Shelf Enclosure

Chapter 1 Introduction 1-9

Ethernet

②

③④

⑤

1-10 Netra CP3260 Blade Server User’s Guide • April 2009

⑥⑦

Figure Legend

1 Remote server

2 Sun Netra CP32X0 Advanced RTM (installed from rear)

3 Serial connection

4 Console terminal

5 Ethernet connection (RJ-45)

6 Sun Netra CP3260 blade server (installed from front)

7 Netra CT 900 server ATCA shelf

1.4 Advanced Rear Transition Modules

An optional Sun Netra CP32X0 Advanced Rear Transition Module (ARTM) can be installed into the rear of the ATCA enclosure, opposite the Netra CP3260 blade server ( server’s Zone 3 rear I/O connectors ( information on the Netra CP32x0 ARTMs.

FIGURE 1-4 shows the physical relationship between the blade server, the rear

transition module, and the midplane in a typical ATCA system.

FIGURE 1-3). The Netra CP32x0 ARTM connects to the Netra CP3260 blade

FIGURE 1-2). Contact your Sun representative for

Chapter 1 Introduction 1-11

FIGURE 1-4 Netra CP3260 Blade Server, Midplane, and Netra CP32x0 ARTM

①

②

1-12 Netra CP3260 Blade Server User’s Guide • April 2009

③

+ 114 hidden pages

Sun Microsystems CP3260 User Manual

Specifications and Main Features

Frequently Asked Questions

User Manual

Contents

Figures

Tables

Preface

Introduction

1.1 Overview

1.2 Features

1.3 System Configurations

1.4 Advanced Rear Transition Modules