Sun Microsystems Sun Fire 15K Overview

Sun Fire™15K/12K Systems

Overview

Sun Microsystems, Inc. www.sun.com

Part No. 806-3509-13 May 2006, Revision A

Submit comments about this document at: http://www.sun.com/hwdocs/feedback

Copyright 2006Sun Microsystems,Inc., 4150Network Circle, SantaClara, California95054, U.S.A.All rightsreserved. Sun Microsystems,Inc. hasintellectual property rightsrelating totechnology embodiedin theproduct that is described inthis document.In

particular,and withoutlimitation, theseintellectual property rightsmay includeone ormore ofthe U.S.patents listedat http://www.sun.com/patents andone ormore additionalpatents orpending patentapplications inthe U.S.and inother countries.

This documentand theproduct towhich itpertains are distributedunder licensesrestricting theiruse, copying,distribution, and decompilation. Nopart ofthe productor ofthis documentmay bereproducedin anyform byany meanswithout priorwritten authorizationof Sun andits licensors,if any.

Third-party software, includingfont technology,is copyrightedand licensedfrom Sun suppliers. Parts ofthe productmay bederived from BerkeleyBSD systems,licensed fromthe Universityof California.UNIX isa registered trademarkin

the U.S.and inother countries,exclusively licensedthrough X/OpenCompany, Ltd. Sun, SunMicrosystems, theSun logo,AnswerBook2, docs.sun.com,Sun Fire, SunFireplane interconnect, andSolaris aretrademarks or

registered trademarks of Sun Microsystems,Inc. inthe U.S.and inother countries. All SPARCtrademarks areused underlicense andare trademarks or registered trademarksof SPARCInternational, Inc.in theU.S. andin other

countries. Productsbearing SPARCtrademarks are basedupon anarchitecture developed by Sun Microsystems,Inc. The OPENLOOK andSun™ GraphicalUser Interfacewas developedby SunMicrosystems, Inc.for itsusers andlicensees. Sunacknowledges

the pioneeringefforts ofXerox in researchingand developingthe conceptof visualor graphicaluser interfacesfor thecomputer industry.Sun holds anon-exclusive licensefrom Xerox tothe XeroxGraphical UserInterface, whichlicense alsocovers Sun’slicensees whoimplement OPEN LOOK GUIsand otherwisecomply withSun’s writtenlicense agreements.

U.S. GovernmentRights-Commercial use.Government usersare subject to the Sun Microsystems, Inc.standard licenseagreement and applicable provisionsof theFAR andits supplements.

DOCUMENTATION IS PROVIDED "AS IS" AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANYIMPLIED WARRANTY OFMERCHANTABILITY, FITNESSFOR A PARTICULAR PURPOSEOR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.

Copyright 2006Sun Microsystems,Inc., 4150Network Circle, SantaClara, Californie95054, Etats-Unis.Tous droitsréservés. Sun Microsystems,Inc. ales droits depropriété intellectuelsrelatants à la technologie incorporée dans leproduit quiest décritdans ce

document. Enparticulier,et sansla limitation,ces droits depropriété intellectuelspeuvent inclure unou plusdes brevetsaméricains énumérés à http://www.sun.com/patents et unou lesbrevets plus supplémentairesou lesapplications debrevet enattente dansles Etats-Uniset dans les autrespays.

Ce produitou documentest protégé parun copyrightet distribuéavec deslicences quien restreignent l’utilisation,la copie,la distribution,et la décompilation. Aucunepartie dece produitou documentne peutêtre reproduite sousaucune forme,parquelque moyen que ce soit, sans l’autorisation préalableet écritede Sunet deses bailleursde licence,s’il yena.

Le logicieldétenu pardes tiers,et quicomprend latechnologie relative auxpolices decaractères, estprotégépar uncopyright etlicencié pardes fournisseurs deSun.

Des partiesde ceproduit pourront êtredérivées dessystèmes BerkeleyBSD licenciéspar l’Universitéde Californie.UNIX estune marque déposée auxEtats-Unis etdans d’autrespays etlicenciée exclusivementpar X/OpenCompany, Ltd.

Sun, SunMicrosystems, lelogo Sun,AnswerBook2, docs.sun.com,Sun Fire, SunFireplane interconnect, etSolaris sontdes marquesde fabrique ou desmarques déposéesde SunMicrosystems, Inc. aux Etats-Unis et dans d’autrespays.

Toutes lesmarques SPARC sont utilisées sous licence et sont des marques defabrique oudes marquesdéposées deSPARC International,Inc. aux Etats-Uniset dansd’autres pays.Les produits protantles marques SPARC sont baséssur unearchitecturedéveloppée parSun Microsystems, Inc.

L’interfaced’utilisation graphiqueOPEN LOOKet Sun™a étédéveloppée parSun Microsystems, Inc.pour sesutilisateurs etlicenciés. Sun reconnaît lesefforts de pionniers de Xeroxpour larecherche et le développment du concept des interfaces d’utilisation visuelle ou graphique pour l’industriede l’informatique.Sun détientune licensenon exclusivedo Xeroxsur l’interfaced’utilisation graphiqueXerox,cette licence couvrant égalementles licenciéesde Sunqui mettenten placel’interface d’utilisation graphiqueOPEN LOOKet quien outrese conforment aux licencesécrites deSun.

LA DOCUMENTATION EST FOURNIE "EN L’ÉTAT" ET TOUTES AUTRES CONDITIONS, DECLARATIONS ET GARANTIES EXPRESSES OU TACITES SONT FORMELLEMENTEXCLUES, DANSLA MESUREAUTORISEE PARLA LOIAPPLICABLE, YCOMPRIS NOTAMMENT TOUTE GARANTIE IMPLICITE RELATIVE A LA QUALITE MARCHANDE, A L’APTITUDE A UNE UTILISATION PARTICULIERE OU A L’ABSENCE DE CONTREFAÇON.

Declaration of Conformity xi

Preface xiii

1. Sun Fire 15K/12K Systems Introduction 1–1

1.1 System Boards 1–2

1.1.1 CPU/Memory Boards 1–2

1.1.2 I/O Boards 1–3

1.1.3 System Controller 1–3

1.1.4 Peripherals 1–3

1.2 System Configuration 1–4

1.3 System Interconnects 1–5

1.3.1 Sun Fireplane Interconnect Architecture 1–6

1.3.2 Address Interconnect 1–7

1.3.3 Data Interconnect 1–7

1.4 Dynamic System Domains 1–8

1.5 Reliability, Availability, and Serviceability 1–9

1.5.1 Integrated Circuit Reliability 1–9

1.5.2 Interconnect Reliability 1–9

1.5.3 Fault-Tolerant Redundancy 1–10

iii

1.5.4 Reconfiguration After Failure 1–10

1.5.5 Serviceability 1–10

2. Dynamic System Domains 2–1

2.1 Domain Configurability 2–1

2.2 Domain Protection 2–3

2.3 Domain Fault Isolation 2–3

3. Reliability, Availability, and Serviceability 3–1

3.1 SPARC CPU Error Protection 3–1

3.2 System Interconnect Error Protection 3–3

3.2.1 Address Interconnect Error Protection 3–3

3.2.2 Data Interconnect Error Protection 3–3

3.2.3 Data Interconnect Error Isolation 3–4

3.2.4 Console Bus Error Protection 3–4

3.3 Redundant Components 3–6

3.3.1 Redundant CPU/Memory Boards 3–6

3.3.2 Redundant I/O Boards 3–6

3.3.3 Redundant PCI Cards 3–7

3.3.4 Redundant System Control Boards 3–7

3.3.5 Redundant System Clocks 3–7

3.3.6 Redundant Power 3–7

3.3.7 Redundant Fans 3–8

3.4 Reconfigurable Sun Fireplane Interconnect 3–8

3.5 Automatic System Recovery 3–9

3.5.1 Built-In Self-Test 3–9

3.5.2 Power-On Self-Test 3–9

3.6 System Controller 3–9

3.6.1 Console Bus 3–10

iv Sun Fire™ 15K/12K Systems • May 2006

3.6.2 Environmental Monitoring 3–10

3.7 Concurrent Serviceability 3–11

3.7.1 Dynamic Reconfiguration of System Boards 3–11

3.7.2 System Controller Board Set Removal and Replacement 3–13

3.7.3 Bulk Power Supply Removal and Replacement 3–13

3.7.4 Fan Tray Removal and Replacement 3–13

3.7.5 Remote Service 3–13

4. System Interconnect 4–1

4.1 Data-Transfer Interconnect Levels 4–3

4.2 Address Interconnect 4–5

4.3 Data Interconnect 4–7

4.4 Interconnect Bandwidth 4–9

4.5 Interconnect Latency 4–10

5. System Components 5–1

5.1 Cabinets 5–2

5.1.1 System Power 5–4

5.1.2 System Cooling 5–4

5.2 Centerplanes 5–5

5.2.1 Sun Fireplane Interconnect 5–7

5.3 System Boards 5–7

5.3.1 System Board Set 5–8

5.3.1.1 Expander Board 5–8

5.3.1.2 CPU/Memory Board 5–8

5.3.1.3 Example of System Board Set 5–9

5.3.1.4 PCI Assembly (hsPCI-X/hsPCI+) 5–9

5.3.1.5 MaxCPU Board 5–9

5.3.2 Controller Board Set 5–13

Contents v

Glossary Glossary–1

Index Index–1

vi Sun Fire™ 15K/12K Systems • May 2006

Figures

FIGURE 1-1 Sun Fire 15K/12K Systems 1–2

FIGURE 1-2 Sun Fireplane Interconnects 1–7

FIGURE 2-1 Example of Domain Configuration With Some Split Board Sets 2–2

FIGURE 3-1 CPU Error Detection and Correction 3–2

FIGURE 3-2 Interconnect ECC and Parity Checking 3–6

FIGURE 4-1 Sun Fire 15K/12K Systems Interconnect 4–2

FIGURE 4-2 Sun Fire 15K/12K Systems Data—Transfer Interconnect Levels 4–3

FIGURE 4-3 Address Interconnect Levels 4–6

FIGURE 4-4 Data Interconnect Levels 4–8

FIGURE 5-1 Sun Fire 15K/12K Systems Major Components 5–2

FIGURE 5-2 Sun Fire 15K/12K Systems Cabinet—Front View 5–3

FIGURE 5-3 Sun Fireplane interconnect and Other Components 5–7

FIGURE 5-4 Board Set Block Diagram 5–11

FIGURE 5-5 System Board Set Layout 5–13

FIGURE 5-6 System Controller Board Layout 5–14

vii

viii Sun Fire™ 15K/12K Systems • May 2006

Tables

TABLE 1-1 Sun Fire 15K/12K System Maximum Configuration 1–4

TABLE 1-2 Sun Fire 15K/12K Systems Interconnect Specifications 1–5

TABLE 4-1 Interconnect Levels 4–4

TABLE 4-2 Peak Interconnect Bandwidth 4–9

TABLE 4-3 Pin-to-Pin Latency for Data in Memory 4–10

TABLE 4-4 Pin-to-Pin Latency for Data in Cache 4–11

x Sun Fire™ 15K/12K Systems • May 2006

Declaration of Conformity

Compliance Model Number: 2080 Product Name: Sun Fire 15K/12K System

EMC

European Union

This equipment complies with the following requirements of the EMC Directive 89/336/EEC:

EN55022:1995/CISPR22:1997 Class A EN55024:1998 EN61000-4-2 4 kV (Direct), 8 kV (Air)

EN61000-4-3 3 V/m EN61000-4-4 1.0 kV Power Lines, 0.5 kV Signal Lines EN61000-4-5 1 kV Line-Line, 2 kV Line-Gnd Power Lines EN61000-4-6 3 V EN61000-4-8 3 A/m

EN61000-4-11 Pass EN61000-3-2:1995 Pass EN61000-3-3:1995 Pass

Safety

This equipment complies with the following requirements of the Low Voltage Directive 73/23/EEC:

EC Type Examination Certificates:

EN60950:1992, 2nd Edition, Amendments 1,2,3,4,11 TÜV Product Service Certificate No.

IEC 950:1991, 2nd Edition, Amendments 1,2,3,4 Evaluated to all CB Countries CB Scheme Certificate No. CB 01 07 17641 014

Z1A 01 07 17641 013

Supplementary Information

This product was tested and complies with all the requirements for the CE Mark.

Dennis P. Symanski DATE Manager, Compliance Engineering

Sun Microsystems, Inc. 4150 Network Circle Santa Clara, CA 95054, USA

Tel: 650-786-3255 Fax: 650-786-3723

Peter Arkless DATE Quality Manager

Sun Microsystems Scotland, Limited Springfield, Linlithgow West Lothian, EH49 7LR Scotland, United Kingdom

Tel: 0506-670000 Fax: 0506 760011

xii Sun Fire™ 15K/12K Systems • May 2006

Preface

This document introduces the Sun Fire™ 15K/12K systems and describes the cabinet, the system, the configuration, the dynamic system domain configurability, the system boards, and the reliability, availability, and serviceability features.

How This Book Is Organized

Chapter 1 describes the systems and boards, the maximum configurations, and the interconnect architecture.

Chapter 2 describes the configurability, inter-domain networking, domain protection, and domain fault isolation.

Chapter 3 defines system error protection, describes the redundant components and system recovery, discusses the system controller technology, and explains the concurrent serviceability features of the systems.

Chapter 4 describes the heart of the system, which is the Sun™ Fireplane interconnect assembly.

Chapter 5 describes the components within the systems.

xiii

Sun Fire 15K/12K Systems Introduction

This chapter provides the following introductory information for the Sun Fire 15K/12K systems:

■ Section 1.1, “System Boards” on page 1-2

■ Section 1.2, “System Configuration” on page 1-4

■ Section 1.3, “System Interconnects” on page 1-5

■ Section 1.4, “Dynamic System Domains” on page 1-8

■ Section 1.5, “Reliability, Availability, and Serviceability” on page 1-9

The Sun Fire 15K/12K systems use the latest UltraSPARC™ III Cu CPU and the Sun Fireplane interconnect architecture running the binary-compatible Solaris™ 8 UNIX operating environment (FIGURE 1-1). The Sun Fireplane interconnect has faster CPUs. The industry-leading dynamic system domain and reliability, availability, and serviceability (RAS) capabilities have been applied and use the active-centerplane technology.

1-1

33.3 in. (84.6 cm)

System Control boards (front and rear)

CPU/Memory boards (9 in front and 9 in rear of the Sun Fire 15K system)

75.5 in.

(191.8 cm)

18 CPU/Memory boards and 18 I/O boards in the Sun Fire 15K system

9 CPU/Memory boards and 9 I/O boards in the Sun Fire 12K system

(with 9 CPU and I/O ﬁller panels in the rear of the system)

FIGURE 1-1 Sun Fire 15K/12K Systems

(9 in front and 9 ﬁller panels in rear of the Sun Fire 12K system)

I/O boards (9 in from and 9 in rear of the Sun Fire 15K system) (9 in front and 9 ﬁller panels in rear of the Sun Fire 12K system)

Bulk power supply (3 in front, 3 in rear)

The Sun Fire 15K/12K systems are essentially the same. The Sun Fire 15K system has the capacity for 18 CPU/Memory boards and 18 I/O boards. The Sun Fire 12K system has the capacity for nine CPU/Memory boards and nine I/O boards. Each system contains two System Control boards (one main and one spare).

1.1 System Boards

1.1.1 CPU/Memory Boards

The CPU/Memory board holds four CPUs. Each CPU has an associated memory subsystem of eight DIMMs, so memory bandwidth and capacity are both scaled up as CPUs are added. The memory capacity of the board is 32 Gbytes using a 1-Gbyte DIMM. The maximum memory bandwidth inside a board is 9.6 Gbytes per second. The CPU/Memory board has a 4.8 Gbyte per second connection to the rest of the system.

1-2 Sun Fire™ 15K/12K Systems • May 2006

1.1.2 I/O Boards

The Sun Fire 15K/12K hot-swap PCI assembly architecture (hsPCI-X/hsPCI+) has two I/O controllers. Each controller provides one 33-MHz peripheral component interconnect (PCI) bus and three 33/66/90 MHz PCI buses for a total of four on each I/O assembly. Therefore, each I/O assembly has four hot-swap component PCI slots. A Sun Fire I/O assembly has a 2.4 Gbyte/sec connection to the rest of the system.

1.1.3 System Controller

The system controller is the heart of the Sun Fire 15K/12K systems availability and serviceability technology. It configures the system, coordinates the boot process, sets up the dynamic system domains, monitors the system environmental sensors, and handles error detection, diagnosis, and recovery. Two System Control boards are configured into the system to provide redundancy and automatic failover in the event that one board fails.

1.1.4 Peripherals

The Sun Fire 15K/12K systems cabinet does not have room for peripherals, with the exception of the system controller peripherals (DVD-ROM, digital audio tape (DAT) drive, and hard drive). However, more peripheral devices can be configured in additional peripheral expansion racks.

Chapter 1 1-3

1.2 System Configuration

TABLE 1-1 summarizes the maximum configuration of the Sun Fire 15K/12K systems.

TABLE 1-1 Sun Fire 15K/12K System Maximum Configuration

Component 15K Configuration 12K Configuration

CPU/Memory boards 18 9

CPUs 72 36

Number of DIMMs 576 288

Memory capacity (with 1-Gbyte DIMMs) 576 GB 288 GB

Sun Fireplane interconnect Active Active

Repeater boards NA NA

Expander boards 18 9

Domains 18 9

I/O boards (assemblies) 18 9

PCI assembly types hsPCI+ hsPCI+

PCI assembly types hsPCI-X hsPCI-X

PCI slots per assembly 4 4

Maximum PCI slots 72 36

Bulk power supplies 6 6

Power requirements 24 kW 24 kW

System Control boards 2 2

Redundant cooling Yes Yes

Redundant AC input Yes Yes

Enclosure Sun Fire 15K/12K

Systems cabinet

Room in enclosure for peripherals No No

Sun Fire 15K/12K

Systems cabinet

1-4 Sun Fire™ 15K/12K Systems • May 2006

1.3 System Interconnects

TABLE 1-2 summarizes the interconnect capacities of the Sun Fire 15K/12K systems.

TABLE 1-2 Sun Fire 15K/12K Systems Interconnect Specifications

Interconnect Specification

System clock 150 MHz

Coherency protocol Snooping on each board set,

System address interconnect 18 snoopy buses,

CPU/Memory board internal bisection bandwidth

CPU/Memory board off-board data port

I/O board off-board data port

System data interconnect 18 3x3 board set crossbars,

System bisection bandwidth 43 Gbytes/sec

Average lmbench (back-to-back-load) latency assumes random accesses

directory across a centerplane

18x18 global address crossbar, 18x18 global response crossbar,

4.8 Gbytes/sec

2.4 Gbytes/sec

18 x 18 global crossbar

326 ns

Note – The definition of snooping, as defined in the PCI System Architecture, Third

Edition, Appendix A: Glossary, 1995, by MindShare, Inc., (ISBN 0-201-40993-3):

Snooping – When a memory access is performed by an agent other than the

cache controller, the cache controller must snoop the transaction to determine if the current master is accessing information that is also resident within the cache. If a snoop hit occurs, the cache controller must take an appropriate action to ensure the continued consistency of its cached information.

Chapter 1 1-5

1.3.1 Sun Fireplane Interconnect Architecture

The Sun Fire 15K/12K systems use the Sun Fireplane interconnect systeminterconnect architecture that is the coherent shared-memory protocol used by the UltraSPARC III Cu CPU generation. This is the fourth generation of shared-memory interconnect. Sun Microsystems uses an improved system interconnect with each new CPU generation to keep system performance scaling with CPU performance.

The Sun Fireplane interconnect architecture is an evolutionary improvement over the previous generation Ultra Port Architecture (UPA). The system clock rate is increased by 50% from 100 MHz to 150 MHz. The snoops per clocks are doubled from one half to one. Taken together, this triples the snooping bandwidth to 150 million addresses per second.

The Sun Fireplane interconnect architecture also adds a new layer of point-to-point directory-coherency protocol. This protocol is used in systems that require more bandwidth than a single snoopy bus can provide. This facility enables coherency to be maintained between multiple snoopy buses.

FIGURE 1-2 shows the Sun Fireplane interconnect architecture of the Sun Fire 15K

system. The board diagrams show the actual on-board connectivity but omit the switch and controller chips for clarity.

18 Sun Fire 15K system

expander boards

PCI

FIGURE 1-2 Sun Fireplane Interconnects

1-6 Sun Fire™ 15K/12K Systems • May 2006

18 x 18 address and response crossbars

Diagram shown: Sun Fire 15K system

PCI

The Sun Fire 15K/12K systems use an expander board to implement a 3x3 switch between a CPU/Memory board, an I/O board, and the Sun Fireplane interconnect port. The Sun Fire 15K/12K systems have three 18x18 crossbars on its Sun Fireplane interconnect for addresses, responses, and data so that address traffic does not interfere with data traffic. The peak Sun Fire 15K/12K systems Sun Fireplane interconnect bandwidth is 43 Gbytes per second.

1.3.2 Address Interconnect

The dashed lines in FIGURE 1-2 are the snoopy address buses. A snoop can occur at every system clock. In the Sun Fire 15K/12K systems, there is a separate snoopy address bus on each board set. A board set is the combination of a CPU/Memory board, an I/O board, and an expander board. Coherency is maintained between board sets by using the point-to-point (directory) portion of the coherency protocol.

1.3.3 Data Interconnect

The solid lines in FIGURE 1-2 represent the data paths. The small circles at the intersections of these lines indicate three-port switches. The CPU/Memory board has three levels of 3x3 switches between a CPU or memory unit and the off-board port. The off-board bandwidth of a CPU/Memory board is 4.8 Gbytes per second. The bandwidth of an I/O board is 2.4 Gbytes per second.

Chapter 1 1-7

1.4 Dynamic System Domains

Each domain in the Sun Fire 15K/12K systems include one or more CPU/Memory boards and one or more I/O boards. Each domain runs its own instance of the Solaris operating environment and has its own peripherals and network connections. Domains can be reconfigured without interrupting the operation of other domains. Domains can be used for:

■ Testing new applications

■ Making operating system updates

■ Supporting various departments

■ Removing and reinstalling boards for repair or upgrade

As an example, the Sun Fire 15K system is divided into three domains. Here is one example of partitioning a fully populated system into three domains to handle three types of functions:

■ Domain 1 is set up to run online transaction processing (OLTP). It is a 32-CPU

domain containing eight boards of four CPUs each.

■ Domain 2 is set up to run decision support software (DSS). It is also a 32-CPU

domain containing eight boards of four CPUs each.

■ Domain 3 is set up as a domain for developers. It is a two-board domain, each

board with four CPUs.

Boards can be automatically migrated between domains as the load change demands.

The Sun Fire 15K system can have up to 18 domains. The Sun Fire 12K system can have up to 9 domains. Domains are isolated from each other by the interconnect application-specific integrated circuits (ASICs).

1-8 Sun Fire™ 15K/12K Systems • May 2006

+ 56 hidden pages