IBM POWER7 User Manual

POWER7

Technology Insight

Wayne Huang

Fang Shu Xin

IBM CONFIDENTIAL – FOR IBM AND BP USE ONLY – NOT FOR DISCLOSURE TO CUSTOMERS

IBM Power Systems

Agenda….

POWER7 Product Family POWER7 Processor Active Memory Expansion POWER7 TPMD

OS Support on POWER7 POWER7 Servers

Power 750

Power 755

Power 770

Power 780

RAS Update

I/O Update

Summary

Power your planet

© 2010 IBM Corporation

IBM Power Systems

POWER7 System Highlights

Balance System Design

 Cache, Memory, and IO

POWER7 Processor Technology

6th Implementation of multi-core design

On chip L2 & L3 caches

POWER7 System Architecture

Blades to High End offerings

Enhances memory implementation

PCIe, SAS / SATA

Built in Virtualization

Memory Expansion

VM Control

Green Technologies

Processor Nap & Sleep Mode

Memory Power Down support

Aggressive Power Save / Capping Modes

Availability

Processor Instruction Retry

Alternate Process Recovery

Concurrent Add & Services

Power your planet

700
600
500
400
300
200
100
0
JS23	JS43	520	550	750	560	570/16	570/32	770	780	595

© 2010 IBM Corporation

IBM Power Systems

Power 2010

Power7

Power 750 (8,16,32 Core)

Power 755 (32 Core) for HPC

Power 770 (12,24,36,48 Core)

Power 780 (16,32,48,64 Core)

POWER6 2010

Power 520, Blades

Power 550

Power 560

Power 570

Power 575	Power 750
Power 595

Power 520

Power Blades

Power your planet

Power 7 Systems

Power 780

Power 770

Power 6 Systems

Power 755

Power 595

Power 570

Power 575

Power 560

Power 550

© 2010 IBM Corporation

IBM Power Systems

POWER7

Pushing the

Limits

Power your planet

© 2010 IBM Corporation

IBM Power Systems

Challenge: Beating Physics to Realize Multi-core Potential

Need to Amplify Effective	Compute Throughput Potential
Socket Throughput
to Close Gap and
Achieve Potential

Socket Throughput Limitation

(Physical signal economics)

Power your

© 2010 IBM Corporation

IBM Power Systems

Trends in Server Evolution

Time

Single Image

Enabled by:

-Technology

-Innovation

Driven by:

-IT Evolution

-Economics

Traditional Entry Server

Single Image Platform

Virtualized/Cloud

Emerging Entry Server

Virtualized/Cloud Platform

8-core 8-core

-A simple matter of riding the multi-core trend?

-Add more cores to the die, beef up some interfaces, and scale to a large SMP?

8-core 8-core

2 to 4 socket

16 to 32-way SMP Server

Traditional High-End Server

Virtualized Consolidation Platform

2-core 2-core

2-core

2-core

2 to 4 socket

8 to 32 socket

Power 4 to 8-way SMP Server

16 to 64-way SMP Server

*Statements regarding SMP servers

do not imply that IBM will introduce a system©with2010thisIBMcapabilityCorporation.

IBM Power Systems

Time

Trends in Server Evolution

Single Image

Enabled by:

-Technology

-Innovation

Driven by:

-IT Evolution

-Economics

Traditional Entry Server

Single Image Platform

2-core 2-core

2-core 2-core

2 to 4 socket

4 to 8-way SMP Server

Virtualized/Cloud

Emerging Entry Server

Virtualized/Cloud Platform

8-core 8-core

8-core

2 to 4

16 to 32-way

-A simple matter of riding the multi-core trend?

-Add more cores to the die, beef up some interfaces, and scale to a large SMP?

Not so simple:

-Emerging entry servers have characteristics similar to traditional high-end large SMP servers

Server

Platform

Achieving solid virtual machine performance requires a Balanced System Structure.

8 to 32 socket

16 to 64-way SMP Server * Statements regarding SMP servers do not imply that IBM will introduce

a system©with2010thisIBMcapabilityCorporation.

IBM Power Systems

Time

Trends in Server Evolution

Single Image

Enabled by:

-Technology

-Innovation

Driven by:

-IT Evolution

-Economics

Traditional Entry Server

Single Image Platform

2-core 2-core

2-core 2-core

2 to 4 socket

4 to 8-way SMP Server

Virtualized/Cloud		UltraScale Cloud
Emerging Entry Server		Emerging High-End Server
Virtualized/Cloud Platform		UltraScale Cloud Platform
8-core	8-core

8-core

8-core

2 to 4 socket

8 to 32 socket

16 to 32-way SMP Server

64 to 256-way SMP Server

Traditional High-End Server

Same enablers and

Virtualized Consolidation Platform

driving factors apply

at larger scale

8 to 32 socket

16 to 64-way SMP Server * Statements regarding SMP servers do not imply that IBM will introduce

a system©with2010thisIBMcapabilityCorporation.

IBM Power Systems

Challenge: How does POWER7 maintain the Balance?

Need to Amplify Effective Socket Throughput to Close Gap and Achieve Potential

Compute Throughput Potential

Cache Hierarchy Technology

and Innovation

Socket Throughput Limitation

(Physical signal economics)

Power your

© 2010 IBM Corporation

IBM Power Systems

Challenge: How does POWER7 maintain the Balance?

Need to Amplify Effective Socket Throughput to Close Gap and Achieve Potential

Compute Throughput Potential

Advances in Memory Subsystem

Cache Hierarchy Technology

and Innovation

Socket Throughput Limitation

(Physical signal economics)

Power your

© 2010 IBM Corporation

IBM Power Systems

Challenge: How does POWER7 maintain the Balance?

Need to Amplify Effective Socket Throughput to Close Gap and Achieve Potential

Compute Throughput Potential

Advances in Off-Chip Signaling

Technology

Advances in Memory Subsystem

Cache Hierarchy Technology

and Innovation

Socket Throughput Limitation

(Physical signal economics)

Power your

© 2010 IBM Corporation

IBM Power Systems

Challenge: How does POWER7 maintain the Balance?

Need to Amplify Effective Socket Throughput to Close Gap and Achieve Potential

Compute Throughput Potential

Exploit Long Term Investment

in Coherence Innovation

Advances in Off-Chip Signaling

Technology

Advances in Memory Subsystem

Cache Hierarchy Technology

and Innovation

Socket Throughput Limitation

(Physical signal economics)

Power your

© 2010 IBM Corporation

IBM Power Systems

POWER7

Processor

Power your planet

© 2010 IBM Corporation

IBM Power Systems
20+ Years of POWER Processors						45nm
					65nm	Next Gen.
		RS64IV Sstar			130nm	POWER7
						-Multi-core
		RS64III Pulsar		180nm	POWER6TM
					-Ultra High Frequency
		RS64II North Star	.18um
			.25um		POWER5TM
	RS64I Apache		.35um		-SMT
	BiCMOS		.5um		Major POWER® Innovation
		.5um		POWER4TM
Muskie A35			.22um	-Dual Core	-1990 RISC Architecture
-Cobra A10	.5um				-1994 SMP
					-1995 Out of Order Execution
-64 bit					-1996 64 Bit Enterprise Architecture
		.35um	POWER3TM		-1997 Hardware Multi-Threading
			-630		-2001 Dual Core Processors
					-2001 Large System Scaling
					-2001 Shared Caches
.72um		POWER2TM			-2003 On Chip Memory Control
		P2SC			-2003 SMT
			.25um		-2006 Ultra High Frequency
RSC		.35um			-2006 Dual Scope Coherence Mgmt
					-2006 Decimal Float/VSX
1.0um		.6um
					-2006 Processor Recovery/Sparing
			604e
					-2009 Balanced Multi-core Processor
		-603
POWER1					-2009 On Chip EDRAM
-AMERICA’s		-601

1990

1995

2000

2005

2010

Power your planet

* Dates represent approximate processor power-on©dates,2010notIBMsystemCorporationavailability

IBM Power Systems

IBM risc processors have many innovations..

			pSeries p640,
64bit				p610
P2,P3,P4		Power3		Power3-II	POWER4	POWER4	POWER4
		200+		333 / 375 / 450	1.1+GHz	1.5GHz	1.8GHz
64bit	RS64	RS64-II	RS64-II	RS64-III	RS64-IV	Regatta
	Apache	Northstar	Northstar	Pulsar	Sstar
	125	262.5	340	450	600+ / 750
32bit				pSeries p620, p660, & p680
		604e				7450
	332 / 375				800MHz/1.0GHz
		1998	1999	2000	2001	2002	2003

Copper =& SOI = & low-k =

Power your planet	Σ
	© 2010 IBM Corporation

IBM Power Systems

Processor Technology Roadmap

POWER5

130 nm

POWER4

180 nm

Dual Core

Chip Multi Processing

Distributed Switch

Shared L2

Dynamic LPARs (32)

Dual Core

Enhanced Scaling

SMT

Distributed Switch +

Core Parallelism +

FP Performance +

Memory bandwidth +

Virtualization

						POWER8
			POWER7
			45 nm
	POWER6
	65 nm

Dual Core

High Frequencies

Virtualization +

Memory Subsystem +

Altivec

Instruction Retry

Dyn Energy Mgmt

SMT +

Protection Keys

Multi Core

On-Chip eDRAM

Power Optimized Cores

	 Mem Subsystem ++	 Concept Phase
	 SMT++
	 Reliability +
	 VSM & VSX (AltiVec)
	 Protection Keys+

2001

2004

2007

2010

Power your planet

© 2010 IBM Corporation

IBM Power Systems

Processor Designs

contrast: 0.278nm H2O

			POWER5	POWER5+	POWER6	POWER7
	Technology		130 nm	90 nm	60 nm	45 nm
	Size		389 mm2	245 mm2	341 mm2	567 mm2
	Transistors		276 M	276 M	790 M	1.2 B
	Cores		2	2	2	4 / 6 / 8
	Frequencies		1.65 GHz	1.9 GHz	3-5 GHz	3-4 GHz
	L2 Cache		1.9 MB Shared	1.9 MB Shared	4 MB / Core	256 KB / Core
	L3 Cache		36 MB	36 MB	32 MB	4 MB / Core
	Memory Cntrl		1	1	2 / 1	2
	LPAR		10 / Core	10 / Core	10 / Core	10 / Core
	Power your		planet			© 2010 IBM Corporation

IBM Power Systems

POWER6 / POWER7

M E M O R Y

POWER6

	Alti	SMT	SMT	Alti
	Vec	Core	Core	Vec
L3	L3	4MB	4MB	L3		L3	M
		L2	L2
	Dir			Dir			E
					Mem
	Mem	Bus Fabric Controller					M
	Ctrl				Ctrl
							O
		GX Bus Cntrl
							R
		GX+ Bridge					Y
		Chip	Chip
		to Chip	to Chip

Power your planet

© 2010 IBM Corporation

IBM Power Systems

POWER6 / POWER7

POWER67

M E M O R Y

Alti	SMT	SMT	Alti
Vec	Core	Core	Vec

	L3 Cache
L3	4MB	4MB	L3		L3	M
Dir	L2	L2
			Dir			E
				Mem
Mem	Bus Fabric Controller					M
Ctrl				Ctrl
						O
	GX Bus Cntrl
						R
	GX+ Bridge					Y
	Chip	Chip
	to Chip	to Chip

eDRAM (Embedded Dynamic RAM)

	 L3 — 6:1 latency improvement (vs. external L3) and 2x BW improvements
		Capacitor vs transister
	 1/3 space (vs 6Trn SRAM cell), 1/5 standby power of standard SRAM
	 Soft error rated 250x lower than SRAM
Power your planet	 Savings of ~ 1.5B transistors over other RAM
		© 2010 IBM Corporation

IBM Power Systems

POWER6 / POWER7

POWER7

P	SMT		SMT		SMT	SMT
WO	Core		Core		Core	Core
	Alti	L2	SMT	L2	SMT	Alti
E			Core		L2	L2
R Vec					Core	Vec

		G		L3 Cache
		X
	M	BL3	4MB		L2	4MB	L3		M
		U	L2			L2	L2
	E	Dir	SMT	L2 SMT		L2	Dir
		S				SMT	SMT		E
			Core	Core		Core	Core
	M	Mem	Bus Fabric Controller					Mem	M
		Ctrl						Ctrl
	O								O
				GX Bus Cntrl
	R								R
	Y			GX+ Bridge					Y
			Chip			Chip
			to Chip			to Chip
		Footprints of working set:
		Private footprints automatically migrates to fast local region 4m/core
		Shared data automatically cloned to multiple private regions

Power your planet

© 2010 IBM Corporation

IBM Power Systems

POWER7 Processor Chip Cores : 8 ( 4 / 6 core options )

Local SMP Links

POWER7

CORE

L2 Cache

MC0

L2 Cache

POWER7

CORE

	F	POWER7
		CORE
	A
	S
	T	L2 Cache

POWER7

CORE

L2 Cache

L3 REGION

L3 Cache and

Chip Interconnect

L2 Cache

POWER7

CORE

L2 Cache

POWER7

CORE

POWER7

CORE

L2 Cache

MC1

L2 Cache

POWER7

CORE

Remote SMP & I/O Links

567mm2 Technology:

 45nm li’thography, Cu, SOI, eDRAM

Transistors: 1.2 B

Equivalent function of 2.7B

eDRAM efficiency

Eight processor cores

12 execution units per core

4 Way SMT per core – up to 4 threads per core

32 Threads per chip

L1: 32 KB I Cache / 32 KB D Cache

L2: 256 KB per core

L3: Shared 32MB on chip eDRAM

Dual DDR3 Memory Controllers

 90 GB/s Memory bandwidth per chip

Binary Compatibility with POWER6

Scalability up to 32 Sockets

360 GB/s SMP bandwidth/chip

20,000 coherent operations in flight

Power your planet

4th Generation SMP Fabric Bus

3rd Generation Multi-Threading

Energy Optimized Design

Enhanced GX System Buses

DDR3 memory

On-chip eDRAM L3

© 2010 IBM Corporation

IBM Power Systems

Cache Hierarchy Technology and Innovation

Core

L2 Cache

Mem Ctrl

L2 Cache

Core

Power your planet

Core	Local	Core
	SMP
L2 Cache	Links	L2 Cache

L3 Cache and Chip Interconnect

		Remote
	L2 Cache		L2 Cache
		+ SMP
	Core		Core
		Links I/O

Core

L2 Cache

Mem Ctrl

L2 Cache

Core

© 2010 IBM Corporation

IBM Power Systems

Cache Hierarchy Technology and Innovation

Core

L2 Cache

Mem Ctrl

L2 Cache

Core

Power your planet

Core	Local	Core
	SMP
L2 Cache	Links	L2 Cache
Fast Local
L3 Region

L3 Cache and Chip Interconnect

		Remote
	L2 Cache		L2 Cache
		+ SMP
	Core		Core
		Links I/O

Core

L2 Cache

Mem Ctrl

L2 Cache

Core

© 2010 IBM Corporation

IBM Power Systems

POWER7 Core

64-bit PowerPC architecture v2.07 Execution Units

•2 Fixed Point Units

•2 Load Store Units

•4 Double Precision Floating Point Units

•1 Branch

•1 Condition Register

•1 Vector Unit

•1 Decimal Floating Point Unit

•6 Wide Dispatch

•Units include distributed Recovery Function

Out of Order Execution excellent for commercial load

Modes: POWER6, POWER6+ and POWER7

e.g. partition mobility+

DFU	VSX
	FPU
ISU
FXU
IFU
CRU/BRU
LSU
L2 Cache

POWER7 continues to support VMX / Extends SIMD support with VSX

 2 VSX (Vec Scalar Ext) units that can each handle 2 Double-Precision FP instructions

8 FLOPS per cycles

VSX units can also handle 4 Single Precision instructions per cycle

VSX instruction set support for vector and scalar instructions

Power your planet

© 2010 IBM Corporation

IBM Power Systems

Advances in Memory Subsystem

Memory Subsystem Requirement

for POWER Servers

Core

Need 10 to 20 GB/s Sustained bandwidth

per Core

Need 16 to 32 GB of Storage per Core

Challenge

for Multi-core POWER7

Socket Challenge:

4x growth in memory bandwidth and capacity needed per socket.

System Challenge:

Packaging more memory into similar volume with

similar energy and cooling constraints.

Energy Constraints

Power your planet

© 2010 IBM Corporation

IBM Power Systems

Advances in Memory Subsystem

Multi-faceted Solution

	POWER7 Chip		1) Dual Integrated DDR3 Controllers
			- Massive 16KB scheduling window
			per POWER7 chip insures high
			channel and DIMM utilization
			- Sparse access acceleration
	Memory	Memory	- Advanced Energy Management
			- Numerous RAS advances
	Controller	Controller

2) Eight high speed 6.4 GHz channels

- New low power differential signaling - Sustained 100+ GB/s per socket

Advanced	3) New DDR3 buffer chip architecture
Buffer
Chip	- Larger capacity support (32 GB / core)
	- Energy Management support
	- RAS enablementh

4) DDR3 DRAMs

- Supports 800, 1066, 1333, and 1600

Power

your

planet

© 2010 IBM Corporation

* Statements regarding memory subsystem features do not imply that IBM will introduce a system with these capabilities.

IBM Power Systems

Memory Channel Bandwidth Evolution

POWER5

POWER6

POWER7

Memory Performance:

Memory Performance:

Memory Performance:

2x DIMM

4x DIMM

6x DIMM

D

D

D

D

D

D

D

D

D

D

D

D

D

D

D

D

D

D

D

D

R

R

R

R

R

R

R

R

R

R

3

3

3

3

3

3

3

3

3

3

DDR2 @ 553 MHz

DDR2 @ 553 / 667 MHz

DDR3 @ 1066 MHz

Effective Bandwidth:

Effective Bandwidth:

Effective Bandwidth:

1.1 GB/s

2.6 GB/sec

6.4 GB/sec

Power your planet

© 2010 IBM Corporation

IBM Power Systems

Exploit Long Term Investment in Coherence Innovation

	Core		Core	Local	Core		Core
				SMP
	L2 Cache		L2 Cache	Links	L2 Cache		L2 Cache
Mem Ctrl		L3 Cache and Chip Interconnect					Mem Ctrl
				Remote
	L2 Cache		L2 Cache		L2 Cache		L2 Cache
				+ SMP
	Core		Core		Core		Core
				Links I/O

Using local and remote SMP links, up to 32 POWER7 chips are connected

Power your planet

© 2010 IBM Corporation

IBM Power Systems

Exploit Long Term Investment in Coherence Innovation

Up to 32 POWER7 chips form a massive SMP system.

Power your planet

*Statements regarding SMP servers do not imply that IBM will introduce

a system with this capability.

© 2010 IBM Corporation