IBM Power Systems E870, Power Systems E880 Technical Overview And Introduction

Front cover

Draft Document for Review October 14, 2014 10:19 am REDP-5137-00

IBM Power Systems E870 and E880

Technical Overview and Introduction

New modular architecture for
increased reliability

Enterprise POWER8
processor-based servers

Exceptional memory and I/O
bandwidth

ibm.com/redbooks

Alexandre Bicas Caldeira

YoungHoon Cho

James Cruickshank

Bartłomiej Grabowski

Redpaper

Draft Document for Review October 14, 2014 10:19 am 5137edno.fm

International Technical Support Organization

IBM Power Systems E870 and E880 Technical Overview
and Introduction

October 2014

REDP-5137-00

5137edno.fm Draft Document for Review October 14, 2014 10:19 am

Note: Before using this information and the product it supports, read the information in “Notices” on

page vii.

First Edition (October 2014)

This edition applies to the IBM Power E870 (9119-MME) and Power E880 (9119-MHE) Power Systems
servers.

This document was created or updated on October 14, 2014.

© Copyright International Business Machines Corporation 2014. All rights reserved.

Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule
Contract with IBM Corp.

Draft Document for Review October 14, 2014 10:19 am 5137TOC.fm

Contents

Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii

Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

Authors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

Now you can become a published author, too! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .x

Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .x

Stay connected to IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi

Chapter 1. General description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.1 Systems overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.1.1 Power E870 server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.1.2 Power E880 server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.1.3 System Control Unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.1.4 System nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.1.5 I/O drawers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

1.2 Operating environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

1.3 Physical package . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

1.4 System features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

1.4.1 Power E870 system features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

1.4.2 Power E880 system features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

1.4.3 Minimum features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

1.4.4 Power supply features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

1.4.5 Processor card features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

1.4.6 Summary of processor features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

1.4.7 Memory features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

1.4.8 System node PCIe slots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

1.5 Disk and media features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

1.6 I/O drawers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

1.6.1 PCIe Gen3 I/O expansion drawer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

1.6.2 I/O drawers and usable PCI slot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

1.6.3 EXP24S SFF Gen2-bay Drawer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

1.7 Comparison between models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

1.8 Build to order . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

1.9 IBM editions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

1.10 Model upgrades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

1.10.1 Power E870 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

1.10.2 Power E880 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

1.10.3 Upgrade considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

1.11 Management consoles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

1.12 System racks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

1.12.1 IBM 7014 model T00 rack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

1.12.2 IBM 7014 model T42 rack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

1.12.3 IBM 7953 model 94Y rack. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

1.12.4 Feature code #0551 rack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

1.12.5 Feature code #0553 rack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

1.12.6 The AC power distribution unit and rack content . . . . . . . . . . . . . . . . . . . . . . . . 26

Chapter 2. Architecture and technical overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

© Copyright IBM Corp. 2014. All rights reserved. iii

5137TOC.fm Draft Document for Review October 14, 2014 10:19 am

2.1 Logical diagrams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

2.2 The IBM POWER8 processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

2.2.1 POWER8 processor overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

2.2.2 POWER8 processor core . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

2.2.3 Simultaneous multithreading. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

2.2.4 Memory access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

2.2.5 On-chip L3 cache innovation and Intelligent Cache . . . . . . . . . . . . . . . . . . . . . . . 42

2.2.6 Level 4 cache and memory buffer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

2.2.7 Hardware transactional memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

2.2.8 Coherent Accelerator Processor Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

2.2.9 Power management and system performance . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

2.2.10 Comparison of the POWER7, POWER7+, and POWER8 processors . . . . . . . . 47

2.3 Memory subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

2.3.1 Custom DIMM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

2.3.2 Memory placement rules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

2.3.3 Memory activation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

2.3.4 Memory throughput . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

2.3.5 Active Memory Mirroring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

2.3.6 Memory Error Correction and Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

2.3.7 Special Uncorrectable Error handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

2.4 Capacity on Demand. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

2.4.1 Capacity Upgrade on Demand (CUoD). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

2.4.2 Power enterprise pools and mobile capacity on demand (Mobile CUoD). . . . . . . 62

2.4.3 Elastic Capacity on Demand (Elastic CoD). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

2.4.4 Utility Capacity on Demand (Utility CoD) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

2.4.5 Trial Capacity on Demand (Trial CoD) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

2.4.6 Software licensing and CoD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

2.5 System bus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

2.5.1 PCI Express Gen3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

2.5.2 Service Processor Bus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

2.6 Internal I/O Subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

2.6.1 Blind-swap cassettes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

2.6.2 System ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

2.7 PCI adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

2.7.1 PCI Express (PCIe). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

2.7.2 LAN Adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

2.7.3 Graphics accelerator adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

2.7.4 SAS adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

2.7.5 Fibre Channel adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

2.7.6 Fibre Channel over Ethernet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

2.7.7 Asynchronous and USB adapters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

2.7.8 InfiniBand host channel adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

2.7.9 Cryptographic Coprocessor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

2.8 Internal Storage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

2.8.1 DVD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

2.9 External I/O subsystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

2.9.1 PCIe Gen3 I/O expansion drawer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

2.9.2 PCIe Gen3 I/O expansion drawer optical cabling . . . . . . . . . . . . . . . . . . . . . . . . . 78

2.9.3 PCIe Gen3 I/O expansion drawer SPCN cabling . . . . . . . . . . . . . . . . . . . . . . . . . 83

2.10 External disk subsystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

2.10.1 EXP24S SFF Gen2-bay Drawer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

2.10.2 EXP24S common usage scenarios. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

2.10.3 IBM System Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

iv IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137TOC.fm

2.11 Hardware Management Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

2.11.1 HMC code level. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

2.11.2 HMC RAID 1 support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

2.11.3 HMC connectivity to the POWER8 processor-based systems . . . . . . . . . . . . . . 98

2.11.4 High availability HMC configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

2.12 Operating system support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

2.12.1 Virtual I/O Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

2.12.2 IBM AIX operating system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

2.12.3 IBM i operating system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

2.12.4 Linux operating system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

2.12.5 Java versions that are supported . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

2.12.6 Boosting performance and productivity with IBM compilers . . . . . . . . . . . . . . . 103

2.13 Energy management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

2.13.1 IBM EnergyScale technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

2.13.2 On Chip Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

2.13.3 Energy consumption estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

Chapter 3. Virtualization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

3.1 IBM POWER Hypervisor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112

3.1.1 Virtual SCSI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

3.1.2 Virtual Ethernet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

3.1.3 Virtual Fibre Channel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114

3.1.4 Virtual (TTY) console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114

3.2 POWER processor modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

3.3 Active Memory Expansion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

3.4 PowerVM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

3.4.1 PowerVM edition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

3.4.2 Logical partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

3.4.3 Multiple shared processor pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122

3.4.4 Virtual I/O Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

3.4.5 PowerVM Live Partition Mobility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

3.4.6 Active Memory Sharing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128

3.4.7 Active Memory Deduplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

3.4.8 Operating system support for PowerVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

3.4.9 Linux support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

3.4.10 PowerVM simplification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

3.4.11 Single view of all Virtual I/O servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

3.4.12 Virtual Storage Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

3.4.13 Virtual Network Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137

3.5 System Planning Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

3.6 IBM Power Virtualization Center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

3.7 IBM Power Virtualization Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140

3.8 VIOS 2.2.3.4 features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

3.9 Dynamic Partition Remote Restart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142

Chapter 4. Reliability, availability, and serviceability. . . . . . . . . . . . . . . . . . . . . . . . . . 145

4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146

4.1.1 RAS enhancements of POWER8 processor-based servers . . . . . . . . . . . . . . . . 146

4.2 Reliability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147

4.2.1 Designed for reliability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147

4.2.2 Placement of components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148

4.3 Processor/Memory availability details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149

4.3.1 Correctable error introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149

Contents v

5137TOC.fm Draft Document for Review October 14, 2014 10:19 am

4.3.2 Uncorrectable error introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150

4.3.3 Processor Core/Cache correctable error handling . . . . . . . . . . . . . . . . . . . . . . . 150

4.3.4 Processor Instruction Retry and other try again techniques . . . . . . . . . . . . . . . . 150

4.3.5 Alternative processor recovery and Partition Availability Priority . . . . . . . . . . . . 151

4.3.6 Core Contained Checkstops and other PowerVM error recovery . . . . . . . . . . . . 151

4.3.7 Cache uncorrectable error handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

4.3.8 Other processor chip functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152

4.3.9 Other fault error handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152

4.3.10 Memory protection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153

4.3.11 I/O subsystem availability and Enhanced Error Handling . . . . . . . . . . . . . . . . . 154

4.4 Enterprise systems availability details. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156

4.5 Availability impacts of a solution architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156

4.5.1 Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157

4.5.2 Virtual I/O redundancy configurations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157

4.5.3 Live Partition Mobility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158

4.6 Serviceability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162

4.6.1 Detecting errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163

4.6.2 Error checkers, fault isolation registers, and First-Failure Data Capture . . . . . . 163

4.6.3 Service processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164

4.6.4 Diagnosing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165

4.6.5 Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166

4.6.6 Notifying . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168

4.6.7 Locating and servicing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169

4.7 Manageability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171

4.7.1 Service user interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171

4.7.2 IBM Power Systems Firmware maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . 176

4.7.3 Concurrent firmware update improvements . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179

4.7.4 Electronic Services and Electronic Service Agent . . . . . . . . . . . . . . . . . . . . . . . 179

4.8 Selected POWER8 RAS capabilities by operating system . . . . . . . . . . . . . . . . . . . . . 183

Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185

IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185

Online resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185

Help from IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186

vi IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137spec.fm

Notices

This information was developed for products and services offered in the U.S.A.

IBM may not offer the products, services, or features discussed in this document in other countries. Consult
your local IBM representative for information on the products and services currently available in your area. Any
reference to an IBM product, program, or service is not intended to state or imply that only that IBM product,
program, or service may be used. Any functionally equivalent product, program, or service that does not
infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to
evaluate and verify the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in this document. The
furnishing of this document does not grant you any license to these patents. You can send license inquiries, in
writing, to:

IBM Director of Licensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785 U.S.A.

The following paragraph does not apply to the United Kingdom or any other country where such
provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION

PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR
IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT,
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of
express or implied warranties in certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodically made
to the information herein; these changes will be incorporated in new editions of the publication. IBM may make
improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time
without notice.

Any references in this information to non-IBM websites are provided for convenience only and do not in any
manner serve as an endorsement of those websites. The materials at those websites are not part of the
materials for this IBM product and use of those websites is at your own risk.

IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring
any obligation to you.

Any performance data contained herein was determined in a controlled environment. Therefore, the results
obtained in other operating environments may vary significantly. Some measurements may have been made
on development-level systems and there is no guarantee that these measurements will be the same on
generally available systems. Furthermore, some measurements may have been estimated through
extrapolation. Actual results may vary. Users of this document should verify the applicable data for their
specific environment.

Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products and cannot confirm the
accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the
capabilities of non-IBM products should be addressed to the suppliers of those products.

This information contains examples of data and reports used in daily business operations. To illustrate them
as completely as possible, the examples include the names of individuals, companies, brands, and products.
All of these names are fictitious and any similarity to the names and addresses used by an actual business
enterprise is entirely coincidental.

COPYRIGHT LICENSE:

This information contains sample application programs in source language, which illustrate programming
techniques on various operating platforms. You may copy, modify, and distribute these sample programs in
any form without payment to IBM, for the purposes of developing, using, marketing or distributing application
programs conforming to the application programming interface for the operating platform for which the sample
programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore,
cannot guarantee or imply reliability, serviceability, or function of these programs.

© Copyright IBM Corp. 2014. All rights reserved. vii

5137spec.fm Draft Document for Review October 14, 2014 10:19 am

Trademarks

IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines
Corporation in the United States, other countries, or both. These and other IBM trademarked terms are
marked on their first occurrence in this information with the appropriate symbol (® or ™), indicating US
registered or common law trademarks owned by IBM at the time this information was published. Such
trademarks may also be registered or common law trademarks in other countries. A current list of IBM
trademarks is available on the Web at http://www.ibm.com/legal/copytrade.shtml

The following terms are trademarks of the International Business Machines Corporation in the United States,
other countries, or both:

Active Memory™
AIX®
Electronic Service Agent™
EnergyScale™
eServer™
Focal Point™
IBM®
Micro-Partitioning®
Power Architecture®
POWER Hypervisor™
Power Systems™

Power Systems Software™
POWER6+™
POWER6®
POWER7+™
POWER7®
PowerHA®
PowerPC®
PowerVM®
POWER®
pSeries®
Rational Team Concert™

Rational®
Redbooks®
Redpaper™
Redbooks (logo) ®
RS/6000®
System p®
System x®
System z®
Tivoli®

The following terms are trademarks of other companies:

Intel Xeon, Intel, Intel logo, Intel Inside logo, and Intel Centrino logo are trademarks or registered trademarks
of Intel Corporation or its subsidiaries in the United States and other countries.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

Microsoft, and the Windows logo are trademarks of Microsoft Corporation in the United States, other
countries, or both.

Java, and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its
affiliates.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Other company, product, or service names may be trademarks or service marks of others.

viii IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137pref.fm

Preface

This IBM® Redpaper™ publication is a comprehensive guide covering the IBM Power System
E870 (9119-MME) and IBM Power System E880 (9119-MHE) servers that support IBM AIX®,
IBM i, and Linux operating systems. The objective of this paper is to introduce the major
innovative Power E870 and Power E880 offerings and their relevant functions:

򐂰 The new IBM POWER8 processor, available at frequencies of 4.024 GHz, 4.190 GHz, and

4.350 GHz

򐂰 Significantly strengthened cores and larger caches
򐂰 Two integrated memory controllers with improved latency and bandwidth
򐂰 Integrated I/O subsystem and hot-pluggable PCIe Gen3 I/O slots
򐂰 Improved reliability, serviceability, and availability (RAS) functions
򐂰 IBM EnergyScale™ technology that provides features such as power trending,

power-saving, capping of power, and thermal measurement

This publication is for professionals who want to acquire a better understanding of IBM Power
Systems™ products. The intended audience includes the following roles:

Authors

򐂰 Clients
򐂰 Sales and marketing professionals
򐂰 Technical support professionals
򐂰 IBM Business Partners
򐂰 Independent software vendors

This paper expands the current set of IBM Power Systems documentation by providing a
desktop reference that offers a detailed technical description of the Power E870 and
Power E880 systems.

This paper does not replace the latest marketing materials and configuration tools. It is
intended as an additional source of information that, together with existing sources, can be
used to enhance your knowledge of IBM server solutions.

This paper was produced by a team of specialists from around the world working at the
International Technical Support Organization, Austin Center.

Alexandre Bicas Caldeira is a Certified IT Specialist and is a member of the Power Systems
Advanced Technical Sales Support team for IBM Brazil. He holds a degree in Computer
Science from the Universidade Estadual Paulista (UNESP) and an MBA in Marketing. His
major areas of focus are competition, sales, and technical sales support. Alexandre has more
than 14 years of experience working on IBM Systems & Technology Group Solutions and has
worked also as an IBM Business Partner on Power Systems hardware, AIX, and IBM
PowerVM® virtualization products.

YoungHoon Cho is a Power Systems Top Gun with the post-sales Technical Support Team
for IBM in Korea. He has over 10 years of experience working on RS/6000®, System p®, and
Power Systems products. He provides second-line technical support to Field Engineers
working on Power Systems and system management.

© Copyright IBM Corp. 2014. All rights reserved. ix

5137pref.fm Draft Document for Review October 14, 2014 10:19 am

James Cruickshank works in the Power System Client Technical Specialist team for IBM in

the UK. He holds an honors degree in Mathematics from the University of Leeds. James has
over 13 years experience working with RS/6000, pSeries®, System p and Power Systems
products. James supports customers in the financial services sector in the UK.

Bartłomiej Grabowski is a Principal System Support Specialist in DHL IT Services in the
Czech Republic. He holds a Bachelor’s degree in Computer Science from the Academy of
Computer Science and Management in Bielsko-Biala. His areas of expertise include IBM i,
PowerHA® solutions that are based on hardware and software replication, Power Systems
hardware, and PowerVM. He is an IBM Certified Systems Expert and a coauthor of several
IBM Redbooks® publications.

The project that produced this publication was managed by:

Scott Vetter

Executive Project Manager, PMP

Thanks to the following people for their contributions to this project:

Tamikia Barrow, Ella Buslovich, Mark Clark, Volker Haug, Daniel Henderson,
Stephanie Jensen, Bob Kovacks, Jai Lei Ma, Chris Mann, Dwayne Moore, Mark Olson,
Monica Sanchez, Bill Starke, Jeff Stuecheli, Doug Szerdi, Jacobo Vargas, Steve Will
IBM

Now you can become a published author, too!

Here’s an opportunity to spotlight your skills, grow your career, and become a published
author—all at the same time! Join an ITSO residency project and help write a book in your
area of expertise, while honing your experience using leading-edge technologies. Your efforts
will help to increase product acceptance and customer satisfaction, as you expand your
network of technical contacts and relationships. Residencies run from two to six weeks in
length, and you can participate either in person or as a remote resident working from your
home base.

Find out more about the residency program, browse the residency index, and apply online at:

ibm.com/redbooks/residencies.html

Comments welcome

Your comments are important to us!

We want our papers to be as helpful as possible. Send us your comments about this paper or
other IBM Redbooks publications in one of the following ways:

򐂰 Use the online Contact us review Redbooks form found at:

ibm.com/redbooks

򐂰 Send your comments in an email to:

[e-mail]

򐂰 Mail your comments to:

IBM Corporation, International Technical Support Organization
Dept. HYTD Mail Station P099

x IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137pref.fm

2455 South Road
Poughkeepsie, NY 12601-5400

Stay connected to IBM Redbooks

򐂰 Find us on Facebook:

http://www.facebook.com/IBMRedbooks

򐂰 Follow us on Twitter:

http://twitter.com/ibmredbooks

򐂰 Look for us on LinkedIn:

http://www.linkedin.com/groups?home=&gid=2130806

򐂰 Explore new Redbooks publications, residencies, and workshops with the IBM Redbooks

weekly newsletter:

https://www.redbooks.ibm.com/Redbooks.nsf/subscribe?OpenForm

򐂰 Stay current on recent Redbooks publications with RSS Feeds:

http://www.redbooks.ibm.com/rss.html

Preface xi

5137pref.fm Draft Document for Review October 14, 2014 10:19 am

xii IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch01.fm

1

Chapter 1. General description

The IBM Power System E870 (9119-MME) and IBM Power System E880 (9119-MHE)
servers use the latest POWER8 processor technology that is designed to deliver
unprecedented performance, scalability, reliability, and manageability for demanding
commercial workloads.

The Power E870 is a highly-scalable rack-mount system optimized for running AIX, IBM i, and
Linux workloads. The Power E870 is a modular-built system and uses one or two system
nodes together with a system control unit. Each system node is five EIA-units tall (5U) and the
system control unit is two EIA-units (2U) tall. The Power E870 is housed in a 19-inch rack.

The Power E880 is a highly-scalable rack-mount system optimized for running AIX, IBM i, and
Linux workloads. The Power E880 is a modular-built system. In 2014, the system is built of
one or two system nodes together with a system control unit. Each system node is five
EIA-units tall (5U) and the system control unit is two EIA-units (2U) tall. The Power E880 is
housed in a 19-inch rack.

© Copyright IBM Corp. 2014. All rights reserved. 1

5137ch01.fm Draft Document for Review October 14, 2014 10:19 am

1.1 Systems overview

The following sections provide detailed information about the Power E870 and Power E880
systems.

1.1.1 Power E870 server

The Power E870 (9119-MME) server is a powerful POWER8-based system that scales up to
eight sockets. Each socket contains a single 8-core or 10-core POWER8 processor. Thus, a
fully-configured Power E870 can scale up to 64 or 80 cores.

The Power E870 is a modular system that is built from a combination of the following four
building blocks:

򐂰 System control unit
򐂰 System node
򐂰 PCIe Gen3 I/O expansion drawer
򐂰 EXP24S SFF Gen2-bay drawer

The system control unit provides redundant system master clock and redundant system
master Flexible Service Processor (FSP) and support for the Operator Panel, the system
VPD, and the base DVD. The system control unit provides clock signals to the system nodes
with semi-flexible connectors.

Each system node provides four processor sockets for POWER8 processors and 32 CDIMM
slots supporting a maximum of 8 memory features. Using the 256 GB memory features, the
system node can support a maximum of 2 TB of RAM. Thus a fully-configured Power E870
can support up to 4 TB of RAM. Each system node provides eight PCIe Gen3 x16 slots.

Each optional 19-inch PCIe Gen3 4U I/O Expansion Drawer provides 12 PCIe Gen 3 slots.
The I/O expansion drawer connects to the system unit with a PCIe X16 to optical CXP
converter adapter housed in the system unit.

Each EXP24S SFF Gen2-bay Drawer provides 24 x 2.5-inch form-factor (SFF) SAS bays.
The EXP24S is connected to the Power E870 server using a SAS adapter in a PCIe slot in a
system node or in a I/O expansion drawer.

2 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch01.fm

Figure 1-1 shows a single system node Power E870 in a T42 rack with two PCIe I/O drawers
and an EXP24S disk drawer.

Figure 1-1 A Power E870 in a T42 rack

1.1.2 Power E880 server

The Power E880 (9119-MHE) server is a powerful POWER8-based system that scales up to
16 sockets. Each socket contains a single 8-core POWER8 processor. Thus, a
fully-configured Power E880 can scale up to 128 cores. In 2014 up to two system nodes per
server are supported or a maximum of 64-cores.

The Power E880 is a modular system that is built from a combination of the following four
building blocks:

򐂰 System control unit
򐂰 System node
򐂰 PCIe Gen3 I/O expansion drawer
򐂰 EXP24S SFF Gen2-bay drawer

The system control unit provides redundant system master clock and redundant system
master Flexible Service Processor (FSP) and support for the Operator Panel, the system
VPD, and the base DVD.

Each system node provides four processor sockets for POWER8 processors and 32 CDIMM
slots supporting a maximum of 8 memory features. Using the 512 GB memory features, the
system node can support a maximum of 4 TB of RAM. Thus a two-node Power E880 can
support up to 8 TB of memory. A four-node Power E880 can support up to 16 TB of RAM.
Each system node provides eight PCIe Gen3 x16 slots.

Chapter 1. General description 3

5137ch01.fm Draft Document for Review October 14, 2014 10:19 am

Each optional 19-inch PCIe Gen3 4U I/O Expansion Drawer provides twelve PCIe Gen 3
slots. The I/O expansion drawer connects to the system unit with a PCIe X16 to Optical CXP
converter adapter housed in the system unit.

Each EXP24S SFF Gen2-bay Drawer provides 24 x 2.5-inch form-factor (SFF) SAS bays.
The EXP24S is connected to the Power E880 server using a SAS adapter in a PCIe slot in a
system node or in a I/O expansion drawer.

Figure 1-2 shows a four system node Power E880 with two PCIe I/O drawers and an EXP24S
disk drawer.

Figure 1-2 A Power E880 in a T42 rack

1.1.3 System Control Unit

The system control unit (SCU) in a Power E870 and E880 is a new innovation to increase the
reliability, availability, and serviceability of the servers. The 2 EIA rack unit (U) unit provides
redundant clock and service processor capability to Power E870 and E880 systems even if
they have only one system unit installed. The SCU also provides a DVD option connected to a
PCIe adapter with a USB cable.

The SCU is powered from the system nodes using cables plugged into the first and second
system nodes in a two-, three-, or four-system node server. The SCU is powered from the
single system node in servers with only one.

The SCU provides redundant clock signalling to the system nodes using semi-flexible
connectors. These connectors are located at the rear of the system and route up and down
the middle of the system nodes. In this way they do not cause any restrictions on the allowed
rack specification.

4 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch01.fm

Power connections

Backup
battery

Clock

connections

DVD USB
connection

FSP

connections

HMC

connections

FSP

connections

HMC

connections

DVD Drive

Operator panel

The SCU provides redundant service processor function to the system nodes using flexible
service processor (FSP) cables. Each system node has two FSP connections to the system
control unit.

The SCU provides redundant connections to one or two HMCs using RJ45 Ethernet
connections.

Figure 1-3 shows the front and rear view of a system control unit. The locations of the
connectors and features are indicated.

Figure 1-3 Front and rear view of the system control unit

1.1.4 System nodes

The system nodes in the Power E870 and E880 servers host the processors, memory, PCIe
slots, and power supplies for the system. Each system node is 5U high and connects to the
system control unit with FSP, clock, and power connectors. The system node connects to
other system nodes using SMP connectors.

Each system node in a Power E870 or E880 server provides four POWER8 processors, 32
CDIMM slots, and eight PCIe Gen3 x16 slots.

Chapter 1. General description 5

5137ch01.fm Draft Document for Review October 14, 2014 10:19 am

PCIe Gen3 x16 slots

Clock connectors

Local clock and control units

Power sockets

FSP connector

SCU power

FSP connector

SCU power

SMP connectors x6

Power sockets Power sockets

Figure 1-4 shows the front view of the system node.

Figure 1-4 Front view of the system node

Figure 1-5 shows the rear view of the system node with notable features highlighted.

Figure 1-5 Rear view of the system node

6 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch01.fm

Unlike the inter-node flex connectors on the Power 770 and Power 780 servers, the SMP
cables on the Power E870 and E880 servers are fully flexible. These cables do not impose
restrictions on allowed rack specification. Figure 1-6 shows a diagram of how SMP cables can
be routed within a rack on a Power E880 with all four system nodes installed.

Figure 1-6 An example of SMP cable routing in a four-drawer system

Figure 1-7 shows SMP cable routing in a two-drawer Power E870 and E880.

Figure 1-7 SMP cable routing in a two-drawer Power E870 and E880.

Chapter 1. General description 7

5137ch01.fm Draft Document for Review October 14, 2014 10:19 am

1.1.5 I/O drawers

I/O drawers to provide additional PCIe adapter slots and SAS disk slots are available for the
Power E870 and E880. More details can be found in section 1.6, “I/O drawers” on page 17.

1.2 Operating environment

Table 1-1details the operating environment for the Power E870 and E880 servers.

Table 1-1 Operating environment for Power E870 and Power E880

Power E870 and Power E880 operating environment

Description Operating Non-operating

Power E870 Power E880 Power E870 Power E880

Temperature 5 to 35 degrees C (41 to 95 F) 5 to 45 degrees C (41 to 114F)

Relative humidity 20 - 80% 8 - 80%

Maximum dew
point

Operating voltage 200 to 240 V AC N/A

Operating
frequency

Maximum power
consumption

Maximum power
source loading

Maximum
thermal output

Maximum altitude 3,048 m (10,000 ft.) N/A

Noise level TBD N/A

1.3 Physical package

Table 1-2 on page 9 lists the physical dimensions of the system control unit and of individual
system nodes. Both servers are available only in a rack-mounted form factor.

29 degrees C (84 F) 28 degrees C (82 F)

47 - 63 Hz N/A

4150 Watts per enclosure N/A

4.2 kVA per enclosure N/A

14,164 BTU/hour per enclosure N/A

The Power E870 is a modular system that can be constructed from a single system control
unit and one or two building-block enclosures.

The Power E880 is a modular system that can be constructed from a single system control
unit and one, two, three, or four building-block enclosures. In 2014, up to two system nodes
per server are supported.

The system control unit requires 2U and each system node requires 5U. Thus, a
single-enclosure system requires 7U, a two-enclosure system requires 12U, a
three-enclosure system requires 17U, and a four-enclosure system requires 22U.

8 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch01.fm

Table 1-2 Physical dimensions of the Power E870 and Power E870

Dimension System control unit Power E870 or

Power E880
enclosure

Width 437 mm (17.2 in) 445 mm (17.5 in) 434 mm (17.1 in)

Depth 784 mm (30.9 in) 902 mm (35.5 in) 813 mm (32.0 in)

PCIe I/O Expansion
drawer

Height 88.9 mm (3.5 in) 2 EIA

units

Weight 23.68 kg (52 lb) 75.7 kg (167 lb) 23.6 kg (52 lb)

219 mm (8.6 in) 5 EIA
units

86 mm (3.4 in) 2 EIA
units

Figure 1-8 shows a picture of a Power E870 system control unit and system node from the
front.

Figure 1-8 Power 870 system control unit and system node

1.4 System features

Both Power E870 and Power E880 system node contain four processor modules with 512 KB
L2 cache and 8 MB L3 cache per core.

1.4.1 Power E870 system features

The following features are available on the Power E870:

򐂰 One or two 5U 19-inch rack mount system node drawers
򐂰 One 2U 19-inch rack-mount system control unit drawer

Chapter 1. General description 9

5137ch01.fm Draft Document for Review October 14, 2014 10:19 am

򐂰 12U for a system with two system node drawers
򐂰 One processor feature per system node. All system nodes must have the same feature:

– 4.024 GHz, (4 X 0/8-core) 32-core POWER8 processor (#EPBA)
– 4.190 GHz, (4 X 0/10-core) 40-core POWER8 processor (#EPBC)

򐂰 Static or mobile processor activation features available on a per core basis
򐂰 POWER8 DDR3 memory CDIMMs (32 CDIMM slots per system node, 16 sites populated

per system node minimum):

– 0/64 GB (4 X 16 GB), 1600 MHz (#EM8J)
– 0/128 GB (4 X 32 GB), 1600 MHz (#EM8K)
– 0/256 GB (4 X 64 GB), 1600 MHz (#EM8L)

򐂰 Active Memory™ Expansion - optimized onto the processor chip (#EM82)
򐂰 Elastic CoD Processor Day - no-charge, temporary usage of inactive, Elastic CoD

resources on initial orders (#EPJ3 or #EPJ5)

򐂰 90 Days Elastic CoD Temporary Processor Enablement (#EP9T)
򐂰 Eight PCIe Gen3 x16 I/O expansion slots per system node drawer (maximum 16 with

2-drawer system)

򐂰 One slim-line, SATA media bay per system control unit enclosure (DVD drive selected as

default with the option to de-select)

򐂰 Redundant hot-swap AC power supplies in each system node drawer
򐂰 Two HMC ports per Flexible Service Processor (FSP) in the system control unit
򐂰 Dynamic logical partition (LPAR) support
򐂰 Processor and memory Capacity Upgrade on Demand (CUoD)
򐂰 PowerVM virtualization:

– Micro-Partitioning®
– Dynamic logical partitioning
– Shared processor pools
– Shared storage pools
– Live Partition Mobility
– Active Memory Sharing
– Active Memory Deduplication
– NPIV support
– PowerVP Performance Monitor

򐂰 Optional PowerHA for AIX, IBM i, and Linux
򐂰 Optional PCIe Gen3 I/O Expansion Drawer with PCIe Gen3 slots:

– Zero or two PCIe Gen3 Drawers per system node drawer (#EMX0)
– Each Gen3 I/O drawer holds two 6-slot PCIe3 Fan-out Modules (#EMXF)
– Each Gen3 I/O drawer attaches to the system node via two PCIe3 Optical Cable

Adapters (#EJ07)

10 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch01.fm

1.4.2 Power E880 system features

The following features are available on the Power E880:

򐂰 One or two 5U 19-inch rack-mount system node drawers
򐂰 One 2U 19-inch rack-mount system control unit drawer
򐂰 12U for a system with two system node drawers
򐂰 One processor feature per system node:

– 4.35 GHz, (4 X 0/8-core) 32-core POWER8 processor (#EPBB)

򐂰 Static or mobile processor activation features available on a per core basis
򐂰 POWER8 DDR3 memory CDIMMs (32 CDIMM slots per system node, 16 sites populated

per system node minimum):

– 0/64 GB (4 X 16 GB), 1600 MHz (#EM8J)
– 0/128 GB (4 X 32 GB), 1600 MHz (#EM8K)
– 0/256 GB (4 X 64 GB), 1600 MHz (#EM8L)
– 0/512 GB (4 X 128 GB), 1600 MHz (#EM8M)

򐂰 Active Memory Expansion - optimized onto the processor chip (#EM82)
򐂰 90 Days Elastic CoD Temporary Processor Enablement (#EP9T)
򐂰 Eight PCIe Gen3 x16 I/O expansion slots per system node drawer (maximum 16 with

2-drawer system)

򐂰 One slim-line, SATA media bay per system control unit enclosure (DVD drive defaulted on

order, option to de-select)

򐂰 Redundant hot-swap ac power supplies in each system node drawer
򐂰 Two HMC ports per Flexible Service Processor (FSP) in the system control unit
򐂰 Dynamic logical partition (LPAR) support
򐂰 Processor and memory CUoD
򐂰 PowerVM virtualization:

– Micro-Partitioning
– Dynamic logical partitioning
– Shared processor pools
– Shared storage pools
– Live Partition Mobility
– Active Memory Sharing
– Active Memory Deduplication
– NPIV support
– PowerVP Performance Monitor

򐂰 Optional PowerHA for AIX, IBM i, and Linux
򐂰 Optional PCIe Gen3 I/O Expansion Drawer with PCIe Gen3 slots:

– Zero or two PCIe Gen3 Drawers per system node drawer (#EMX0)
– Each Gen3 I/O drawer holds two 6-slot PCIe3 Fan-out Modules (#EMXF)
– Each Gen3 I/O drawer attaches to the system node via two PCIe3 Optical Cable

Adapters (#EJ07)

Chapter 1. General description 11

5137ch01.fm Draft Document for Review October 14, 2014 10:19 am

1.4.3 Minimum features

Each Power E870 or Power E880 initial order must include a minimum of the following items:
򐂰 1 x processor module

– 4.024 GHz, 32-core POWER8 processor module (#EPBA) for Power E870
– 4.190 GHz, 40-core POWER8 processor module (#EPBC) for Power E870
– 4.35 GHz, 32-core POWER8 processor module (#EPBB) for Power E880

򐂰 8 x 1 core processor activation

– 1 core Processor Activation for #EPBA (#EPBJ) for Power E870
– 1 core Processor Activation for #EPBC (#EPBL) for Power E870
– 1 core Processor Activation for #EPBB (#EPBK) for Power E880

򐂰 4 x 64 GB (4 x 16 GB) CDIMMs, 1600 MHz, 4 Gb DDR3 DRAM (#EM8J)
򐂰 50% of installed memory must be activated using features:

– 1 GB Memory Activation (#EMA5)
– 100 1 GB Memory Activations (#EMA6)

򐂰 1 x 5U system node drawer (#EBA0)
򐂰 2 x Service Processor (#EU0A)
򐂰 1 x Load Source Specify

– EXP24S SFF Gen2 (#5887 or #EL1S) Load Source Specify (#0728)
– SAN Load Source Specify (#0837)

򐂰 1 x Rack-mount Drawer Bezel and Hardware

– IBM Rack-mount Drawer Bezel and Hardware (#EBA2) for Power E870
– IBM Rack-mount Drawer Bezel and Hardware (#EBA3) for Power E880
– OEM Rack-mount Drawer Bezel and Hardware (#EBA4)

򐂰 1 x System Node to System Control Unit Cable Set for Drawer 1 (#ECCA)
򐂰 1 x AC Power Chunnels for routing power cables from back of machine to front (#EBAA)
򐂰 1 x Language Group Specify (#9300/#97xx)
򐂰 1 x Primary Operating System Indicator

– Primary Operating System Indicator - AIX (#2146)
– Primary Operating System Indicator - Linux (#2147)

򐂰 Optional 1 x Media

– SATA Slimline DVD-RAM with write cache (#EU13)
– PCIe2 LP 4-Port USB 3.0 Adapter (#EC45)
– PCIe2 4-Port USB 3.0 Adapter (#EC46)

When either AIX or Linux are the primary operating systems, the order must include a
minimum of the following items also:

򐂰 1 x Primary Operating System Indicator

– Primary Operating System Indicator - AIX (#2146)
– Primary Operating System Indicator - Linux (#2147)

12 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch01.fm

When IBM i is the primary operating system, the order must include a minimum of the
following items:

򐂰 1 x Specify Code

– Mirrored System Disk Level, Specify Code (#0040)
– Device Parity Protection-All, Specify Code (#0041)
– Mirrored System Bus Level, Specify Code (#0043)
– Device Parity RAID 6 All, Specify Code (#0047)
– Mirrored Level System Specify Code (#0308)

򐂰 1 x System Console

– Sys Console On HMC (#5550)
– System Console-Ethernet No IOP (#5557)

򐂰 1 x Primary Operating System Indicator - IBM i (#2145)

Note:

򐂰 Additional optional features can be added, as desired.
򐂰 IBM i systems require a DVD to be available to the system when required. This DVD

can be located in the system control unit (DVD feature #EU13) or it can be located
externally in an enclosure like the 7226-1U3. A USB PCIe adapter such as #EC45 is
required for #EU13. A SAS PCIe adapter such as #EJ11 is required to attach a SATA
DVD in the 7226-1U3. A virtual media repository may be used to substitute for a DVD
device.

򐂰 Feature-coded racks are allowed for I/O expansion only.
򐂰 A machine type/model rack, if desired, should be ordered as the primary rack.
򐂰 A minimum number of eight processor activations must be ordered per system.
򐂰 A minimum of four memory features per drawer are required.
򐂰 At least 50% of available memory must be activated through a combination of feature

#EMA5 and #EMA6, and #EMA9.

򐂰 Memory sizes can differ across CPU modules, but the eight CDIMM slots connected to

the same processor module must be filled with identical CDIMMs.

򐂰 If SAN Load Source Specify (#0837) is ordered #0040, #0041, #0043, #0047, #0308

are not supported.

򐂰 The language group is auto-selected based on geographic rules.
򐂰 No feature codes are assigned for the following:

– Four AC power supplies are delivered as part of the system node. No features are

assigned to power supplies. Four line cords are auto-selected according to
geographic rules.

– Two default AC PDU to wall cables are included. No features are assigned. Cables

are auto-selected according to geographic rules.

– There must be one system control unit on each system. The system control unit is

considered the system with the serial number.

򐂰 One HMC is required for every Power E870 or Power E880; however, a communal HMC

is acceptable. HMCs supported on POWER8 hardware are 7042-C08 and 7042-CR5
through 7042-CR8.

Chapter 1. General description 13

5137ch01.fm Draft Document for Review October 14, 2014 10:19 am

1.4.4 Power supply features

This section describes how the system nodes and system control units are powered.

System node power

Four AC power supplies are required for each system node enclosure. This arrengement
provides 2+2 redundant power with dual power sources for enhanced system availability. A
failed power supply can be hot swapped but must remain in the system until the replacement
power supply is available for exchange.

Four AC power cords are used for each system node (one per power supply) and are ordered
using the AC Power Chunnel feature (#EBAA). The chunnel carries power from the rear of the
system node to the hot swap power supplies located in the front of the system node where
they are more accessible for service.

System control unit power

The system control unit is powered from the system nodes. UPIC cables provide redundant
power to the system control unit. Two UPIC cables attach to system node drawer 1 and two
UPIC cables attach to system node drawer 2. They are ordered with #ECCA and #ECCB. The
UPIC cords provides N+1 redundant power to the system control unit.

1.4.5 Processor card features

Each system must have a minimum of four active processors. Each processor card feature
will deliver a set of four identical processors. All processor cards in the system must be
identical. Cable features are required to connect system nodes to the system control unit and
to other system nodes.

򐂰 For a single system node configuration, #ECCA is required. #ECCA provides cables to

connect the system node with the system control unit.

򐂰 For a dual system node configuration, #ECCB is required. #ECCB provides cables to

connect the system nodes with the system control unit and cables to connect the two
system nodes.

The Power E870 has two types of processor cards, offering the following features:

򐂰 4.024 GHz, (4 X 0/8-core) 32-core POWER8 processor (#EPBA)
򐂰 4.190 GHz, (4 X 0/10-core) 40-core POWER8 processor (#EPBC)

The Power E880 has one type of processor card, offering the following features:
򐂰 4.35 GHz, (4 X 0/8-core) 32-core POWER8 processor (#EPBB)

Several types of capacity on demand (CoD) processor options are available on the
Power E870 and Power 880 servers to help meet changing resource requirements in an
on-demand environment by using resources installed on the system but not activated. CoD
allows you to purchase additional permanent processor or memory capacity and dynamically
activate it when needed. The #EPJ3 provide no-charge elastic processor days for
Power E880.

1.4.6 Summary of processor features

Table 1-3 on page 15 summarizes the processor feature codes for the Power E870.

14 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch01.fm

Table 1-3 Summary of processor features for the Power E870

Feature Code Description

EPBA 4.02 GHz, 32-core POWER8 processor

EPBC 4.19 GHz, 40-core POWER8 processor

EB2R Single 5250 Enterprise Enablement, IBM i

EB30 Full 5250 Enterprise Enablement, IBM i

EP2S 1-Core Mobile Activation

EP2U 1-Core Mobile Activation from POWER 7

EP9T 90 Days Elastic CoD Processor Core Enablement

EPBJ 1 core Processor Activation for #EPBA

EPBL 1 core Processor Activation for #EPBC

EPBN 1 core Processor Activation for #EPBA, Mobile Enabled

EPBQ 1 core Processor Activation for #EPBC, Mobile Enabled

EPJ6 1 Proc-Day Elastic CoD Billing for #EPBA, AIX/Linux

EPJ7 1 Proc-Day Elastic CoD Billing for #EPBA, IBM i

EPJ8 100 Elastic CoD Proc-Days of Billing for Processor #EPBA. AIX/Linux

EPJ9 100 Elastic CoD Proc-Days of Billing for Processor #EPBA. IBM i

EPJA Proc CoD Utility Billing, 100 Proc-mins. for #EPBA, AIX/Linux

EPJB Proc CoD Utility Billing, 100 Proc-mins. for #EPBA, IBM i

EPJJ 1 Proc-Day Elastic CoD Billing for #EPBC, AIX/Linux

EPJK 1 Proc-Day Elastic CoD Billing for #EPBC, IBM i

EPJL 100 Elastic CoD Proc-Days of Billing for Processor #EPBC. AIX/Linux

EPJM 100 Elastic CoD Proc-Days of Billing for Processor #EPBC. IBM i

EPJN Proc CoD Utility Billing, 100 Proc-mins. for #EPBC, AIX/Linux

EPJP Proc CoD Utility Billing, 100 Proc-mins. for #EPBC, IBM i

Table 1-4 summarizes the processor feature codes for the Power E880.

Table 1-4 Summary of processor features for the Power E880

Feature Code Description

EPBB 4.35 GHz, 32-core POWER8 processor

EB2R Single 5250 Enterprise Enablement

EB30 Full 5250 Enterprise Enablement

EP2T 1-Core Mobile Activation

EP2V 1-Core Mobile Activation from POWER 7

EP9T 90 Days Elastic CoD Processor Core Enablement

EPBK 1 core Processor Activation for #EPBB

Chapter 1. General description 15

5137ch01.fm Draft Document for Review October 14, 2014 10:19 am

Feature Code Description

EPBP 1 core Processor Activation for #EPBB, Mobile Enabled

EPJ3 48 Proc-Days of Elastic CoD Processor Resources

EPJC 1 Proc-Day Elastic CoD Billing for #EPBB, AIX/Linux

EPJD 1 Proc-Day Elastic CoD Billing for #EPBB, IBM i

EPJE 100 Elastic CoD Proc-Days of Billing for Processor #EPBB. AIX/Linux

EPJF 100 Elastic CoD Proc-Days of Billing for Processor #EPBB. IBM i

EPJG Proc CoD Utility Billing, 100 Proc-mins. for #EPBB, AIX/Linux

EPJH Proc CoD Utility Billing, 100 Proc-mins. for #EPBB, IBM i

1.4.7 Memory features

1600 MHz memory CDIMMs are available as 64 GB (#EM8J), 128 GB (#EM8K), 256 GB
(#EM8L), and 512 GB (#EM8M) memory features. Each memory feature provides four
CDIMMs. CDIMMs are custom DIMMs which enhance memory performance and memory
reliability. Each system node has 32 CDIMM slots which support a maximum of eight memory
features. Memory activations of 50% of the installed capacity are required. The feature
#EMJ8 provides no-charge elastic memory days for the Power E880.

Table 1-5 lists memory features that are available for the Power E870 and E880.

Table 1-5 Summary of memory features

Feature Code Description

EM8J 64 GB (4X16 GB) CDIMMs, 1600 MHz, 4 GBIT DDR3 DRAM

EM8K 128 GB (4X32 GB) CDIMMs, 1600 MHz, 4 GBIT DDR3 DRAM

EM8L 256 GB (4X64 GB) CDIMMs, 1600 MHz, 4 GBIT DDR3 DRAM

EM8M 512 GB (4X128 GB) CDIMMs, 1600 MHz, 4 GBIT DDR3 DRAM (E880 only)

EMB6 Bundle of eight #EM8M, 512 GB 1600 MHz Memory features

EM9T 90 Days Elastic CoD Memory Enablement

EMA5 1 GB Memory Activation

EMA6 Quantity of 100 1 GB Memory Activations (#EMA5)

EMA7 100 days of 1 GB Mobile Memory Activations

EMA9 100 days of 1 GB Mobile Enabled Memory Activations

EMB7 Bundle of forty, #EMA6 1600 MHz memory features

EMB8 Bundle of 512 #EMA5 1600 MHz memory features

EMJ4 1 GB-Day billing for Elastic CoD memory

EMJ5 100 GB-Day billing for Elastic CoD memory

EMJ6 999 GB-Day billing for Elastic CoD memory

EMJ8 384 GB-Days of Elastic CoD Memory Resources (E880 only)

16 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch01.fm

1.4.8 System node PCIe slots

Each Power E870 or Power E880 system node enclosure provides eight half-length, half-high
x16 PCIe Gen3 slots. These PCIe slots can be used for either low profile PCIe adapters or for
attaching a PCIe I/O drawer.

A new form factor blind swap cassette (BSC) is used to house the low profile adapters which
go into these slots.

PCIe Gen1, Gen2, and Gen3 adapter cards are supported in these Gen3 slots.

Table 1-6 provides details of the PCI slots in the Power E870 and Power E880 system.

Table 1-6 PCIe slot locations and descriptions for the Power E870 and Power E880

Slot Location code Description PHB Adapter size

Slot 1 P1-C1 PCIe3, x16 Processor Module 1, PHB1 Short (low-profile)

Slot 2 P1-C2 PCIe3, x16 Processor Module 1, PHB0 Short (low-profile)

Slot 3 P1-C3 PCIe3, x16 Processor Module 2, PHB1 Short (low-profile)

Slot 4 P1-C4 PCIe3, x16 Processor Module 2, PHB0 Short (low-profile)

Slot 5 P1-C5 PCIe3, x16 Processor Module 3, PHB1 Short (low-profile)

Slot 6 P1-C6 PCIe3, x16 Processor Module 3, PHB0 Short (low-profile)

Slot 7 P1-C7 PCIe3, x16 Processor Module 4, PHB1 Short (low-profile)

Slot 8 P1-C8 PCIe3, x16 Processor Module 4, PHB0 Short (low-profile)

򐂰 All slots support enhanced error handling (EEH).
򐂰 All PCIe slots are hot swappable and support concurrent maintenance.

1.5 Disk and media features

A system node does not support internal disks. Any required disk must reside within a SAN
disk subsystem or an external disk drawer. At the time of writing, the EXP24S SFF Gen2-bay
Drawer (#5887) is the only supported disk drawer for Power E870 or Power E880.

Each system control unit enclosure has one slim-line bay which can support one DVD drive
(#EU13). The #EU13 DVD is cabled to a USB PCIe adapter located in either a system node
or in a PCIe Gen3 I/O drawer. A USB to SATA converter is included in the configuration
without a separate feature code.

For IBM i, a DVD drive must be available on the server when required. The DVD can be in the
system control unit or it can be located in an external enclosure such as a 7226-1U3
Multimedia drawer.

1.6 I/O drawers

If additional Gen3 PCIe slots beyond the system node slots are required, a system node x16
slot is used to attach a 6-slot expansion module in the I/O Drawer. A PCIe Gen3 I/O

Chapter 1. General description 17

5137ch01.fm Draft Document for Review October 14, 2014 10:19 am

expansion drawer (#EMX0) holds two expansion modules which are attached to any two x16
PCIe slots in the same system node or in different system nodes.

Disk-only I/O drawers (#5887) are also supported, providing storage capacity.

1.6.1 PCIe Gen3 I/O expansion drawer

The 19-inch 4 EIA (4U) PCIe Gen3 I/O expansion drawer (#EMX0) and two PCIe FanOut
Modules (#EMXF) provide twelve PCIe I/O full-length, full-height slots. One FanOut Module
provides six PCIe slots labeled C1 through C6. C1 and C4 are x16 slots and C2, C3, C5, and
C6 are x8 slots. PCIe Gen1, Gen2, and Gen3 full-high adapter cards are supported.

A blind swap cassette (BSC) is used to house the full-high adapters that go into these slots.
The BSC is the same BSC as used with the previous generation server's 12X attached I/O
drawers (#5802, #5803, #5877, #5873). The drawer is shipped with a full set of BSC, even if
the BSC is empty.

Concurrent repair and add/removal of PCIe adapter cards is done by HMC guided menus or
by operating system support utilities.

A PCIe X16 to Optical CXP converter adapter (#EJ07) and 2.0 M (#ECC6) or 10.0 M
(#ECC8) CXP 16X Active Optical cables (AOC) connect the system node to a PCIe FanOut
module in the I/O expansion drawer. One feature #ECC6 or one #ECC8 ships two AOC
cables. Each PCIe Gen3 I/O expansion drawer has two power supplies.

In 2014, a system node supports either zero or two PCIe Gen3 I/O expansion drawers.
Connecting a single drawer to a system node is not supported.

Figure 1-9 shows a PCIe Gen3 I/O expansion drawer.

Figure 1-9 PCIe Gen3 I/O expansion drawer

18 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch01.fm

P1-C3

P1-C2

P1-C1

P1-C4

P1-C5

P1-C6

P2-C1

P2-C2

P2-C3

P2-C6

P2-C5

P2-C4

1.6.2 I/O drawers and usable PCI slot

Figure 1-10 shows the rear view of the PCIe Gen3 I/O expansion drawer with the location
codes for the PCIe adapter slots in the PCIe3 6-slot fanout module.

Figure 1-10 Rear view of a PCIe Gen3 I/O expansion drawer with PCIe slots location codes

Table 1-7 provides details of the PCI slots in the PCIe Gen3 I/O expansion drawer.

Table 1-7 PCIe slot locations and descriptions for the PCIe Gen3 I/O expansion drawer

Slot Location code Description

Slot 1 P1-C1 PCIe3, x16

Slot 2 P1-C2 PCIe3, x8

Slot 3 P1-C3 PCIe3, x8

Slot 4 P1-C4 PCIe3, x16

Slot 5 P1-C5 PCIe3, x8

Slot 6 P1-C6 PCIe3, x8

Slot 7 P2-C1 PCIe3, x16

Slot 8 P2-C2 PCIe3, x8

Slot 9 P2-C3 PCIe3, x8

Slot 10 P2-C4 PCIe3, x16

Slot 11 P2-C5 PCIe3, x8

Slot 12 P2-C6 PCIe3, x8

򐂰 All slots support full-length, regular-height adapter or short (low-profile) with a

regular-height tailstock in single-wide, generation 3, blind-swap cassettes.

Chapter 1. General description 19

5137ch01.fm Draft Document for Review October 14, 2014 10:19 am

1 2 3 4 5 6 7 8 910111213141516 1718192021222324

FC 58 87

C1

C2

LNK LNK LN K LN K

T2 T 3T1

LNK LNK

10/100 LNK

LNK LNK LNK LNK

T2 T3T1

LNK LNK

10/100 LNK

ESMESM

OK

IN

DC

OK

IN

DC

Front

Rear

򐂰 Slots C1 and C4 in each PCIe3 6-slot fanout module are x16 PCIe3 buses and slots C2,

C3, C5, and C6 are x8 PCIe buses.

򐂰 All slots support enhanced error handling (EEH).
򐂰 All PCIe slots are hot swappable and support concurrent maintenance.

Table 1-8 summarizes the maximum number of I/O drawers supported and the total number
of PCI slots that are available when expansion consists of a single drawer type.

Table 1-8 Maximum number of I/O drawers supported and total number of PCI slots

System nodes Maximum #EMX0

drawers

Total number of slots

PCIe3, x16 PCIe3, x8

1 system node 2 8 16

2 system nodes 4 16 32

1.6.3 EXP24S SFF Gen2-bay Drawer

The EXP24S SFF Gen2-bay Drawer (#5887) is an expansion drawer with 24 2.5-inch
form-factor SAS bays. The EXP24S supports up to 24 hot-swap SFF-2 SAS hard disk drives
(HDDs) or solid state drives (SSDs). It uses only 2 EIA of space in a 19-inch rack. The
EXP24S includes redundant AC power supplies and uses two power cords.

With AIX, Linux, and VIOS, you can order the EXP24S with four sets of six bays, two sets of
12 bays, or one set of 24 bays (mode 4, 2, or 1). With IBM i, you can order the EXP24S as
one set of 24 bays (mode 1). Mode setting is done by IBM Manufacturing and there is no
option provided to change the mode after it is shipped.

The EXP24S SAS ports are attached to a SAS PCIe adapter or pair of adapters using SAS
YO or X cables.

Figure 1-11 shows the EXP24S SFF Gen2-bay drawer.

20 IBM Power Systems E870 and E880 Technical Overview and Introduction

Figure 1-11 EXP24S SFF Gen2-bay drawer

Draft Document for Review October 14, 2014 10:19 am 5137ch01.fm

1.7 Comparison between models

The Power E870 offers configuration options where the POWER8 processor can have one of
two processor speeds. Each system node is populated with four single chip modules (SCMs).
Each system node will contain one of the following processor configurations:

򐂰 Four 4.02 GHz 8-core SCMs
򐂰 Four 4.19 GHz 10-core SCMs

A Power E870 with either of the processor configurations can have as few as eight cores
activated or up to 100% of the cores can be activated. Incrementing one core at a time is
available through built-in capacity on demand (CoD) functions to the full capacity of the
system. The Power E870 can be installed with one or two system nodes connected to the
system control unit. Each system node can have up to 2 TB of memory installed.

The Power E880 also offers system nodes populated with four SCMs. The Power E880
system node has one processor configuration:

򐂰 Four 4.35 GHz 8-core SCMs

A Power E880 with either of the processor configurations can have as few as eight cores
activated or up to 100% of the cores can be activated. Incrementing one core at a time is
available through built-in capacity on demand (CoD) functions to the full capacity of the
system. The Power E880 can be installed with one, two, three, or four system nodes
connected to the system control unit. In 2014 up to two system nodes are supported. Each
system node can have up to 4 TB of memory installed.

Statement of direction:

IBM plans to enhance the Power Systems enterprise system portfolio with greater
scalability, availability and flexibility. IBM intends to deliver the following offerings:

򐂰 A more scalable Power E880 enterprise-class system with up to 192 POWER8

processor cores and up to 16 TB of total memory

򐂰 Support for concurrent maintenance on the Power E880 I/O Expansion Drawer by

enabling hot add and repair capabilities

IBM's statements regarding its plans, directions, and intent are subject to change or
withdrawal without notice at IBM's sole discretion. Information regarding potential future
products is intended to outline our general product direction and it should not be relied on
in making a purchasing decision. The information mentioned regarding potential future
products is not a commitment, promise, or legal obligation to deliver any material, code, or
functionality. Information about potential future products may not be incorporated into any
contract. The development, release, and timing of any future features or functionality
described for our products remains at our sole discretion.

Table 1-9 shows a summary of processor and memory maximums for the Power E870 and
Power E880.

Table 1-9 Summary of processor and memory maximums for the Power E870 and Power E880

System Cores

per
SCM

Core
speed

System
node core
maximum

System
core
maximum

System
node
memory
maximum

System
memory
maximum

Power E870 8 4.02 GHz 32 64 2 TB 4TB

Chapter 1. General description 21

5137ch01.fm Draft Document for Review October 14, 2014 10:19 am

System Cores

Power E870 10 4.19 GHz 40 80 2 TB 4 TB

Power E880 in
2014

Power E880 8 4.35 GHz 32 128 4 TB 16TB

1.8 Build to order

You can do a build to order (also called a la carte) configuration by using the IBM
Configurator for e-business (e-config). With it, you specify each configuration feature that you
want on the system.

This method is the only configuration method for the Power E870 and Power E880 servers.

1.9 IBM editions

Core
per
SCM

8 4.35 GHz 32 64 4 TB 8TB

speed

System
node core
maximum

System
core
maximum

System
node
memory
maximum

System
memory
maximum

IBM edition offerings are not available for the Power E870 and Power E880 servers.

1.10 Model upgrades

The following sections describe the various upgrades that are available.

1.10.1 Power E870

A model conversion from a Power 770 (9117-MMD) to a Power E870 (9119-MME) is
available. One-step upgrades from previous models of the Power 770 (9117-MMB and
9117-MMC) are not available. To upgrade from a 9117-MMB or 9117-MMC, an upgrade to a
9117-MMD is required first.

The existing components being replaced during a model or feature conversion become the
property of IBM and must be returned.

1.10.2 Power E880

A model conversion from a Power 780 (9179-MHD) to a Power E880 (9119-MHE) is available.
One-step upgrades from previous models of the Power 780 (9179-MHB and 9179-MHC) are
not available. To upgrade from a 9179-MHB or 9179-MHC, an upgrade to a 9179-MHD is
required first.

Upgrades to a Power E880 from a Power 795 (9119-FHB) are not available.

The existing components being replaced during a model or feature conversion become the
property of IBM and must be returned.

22 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch01.fm

1.10.3 Upgrade considerations

Feature conversions are set up for the following items:

򐂰 PCIe Crypto blind swap cassettes
򐂰 Power IFL processor activations
򐂰 Power IFL PowerVM for Linux
򐂰 Active Memory Expansion Enablement
򐂰 DDR3 memory DIMMS to CDIMMS
򐂰 Static and mobile memory activations
򐂰 5250 enterprise enablement
򐂰 POWER7+™ processor cards to POWER8 processors
򐂰 Static and mobile processor activations
򐂰 System CEC enclosure and bezel to 5U system node drawer
򐂰 PowerVM standard and enterprise

The following features that are present on the current system can be moved to the new
system if they are supported in the Power E870 and E880:

򐂰 Disks
򐂰 SSDs
򐂰 PCIe adapters with cables, line cords, keyboards, and displays
򐂰 Racks
򐂰 Doors
򐂰 EXP24S I/O drawers

For POWER7+ processor-based systems that have the Elastic CoD function enabled, you
must reorder the elastic CoD enablement features when placing the upgrade MES order for
the new Power E870 or E880 system to keep the elastic CoD function active. To initiate the
model upgrade, the on/off enablement features should be removed from the configuration file
before the MES order is started. Any temporary use of processors or memory owed to IBM on
the existing system must be paid before installing the Power E870 or E880.

1.11 Management consoles

This section discusses the supported management interfaces for the servers.

The Hardware Management Console (HMC) is required for managing the IBM
Power E870 and Power E880. It has a set of functions that are necessary to manage
the system:

򐂰 Creating and maintaining a multiple partition environment
򐂰 Displaying a virtual operating system session terminal for each partition
򐂰 Displaying a virtual operator panel of contents for each partition
򐂰 Detecting, reporting, and storing changes in hardware conditions
򐂰 Powering managed systems on and off
򐂰 Acting as a service focal point for service representatives to determine an appropriate

service strategy

Chapter 1. General description 23

5137ch01.fm Draft Document for Review October 14, 2014 10:19 am

In 2014, a new HMC model was announced, machine type 7042-CR8. Hardware features on
the CR8 model include a new x86 processor, dual hard drives (RAID 1), and redundant power
supply. At the time of writing, the latest version of HMC code was V8.2.0. This code level is
required for new the new systems.

The IBM Power E870 and Power E880 are not supported by the Integrated Virtualization
Manager (IVM).

Several HMC models are supported to manage POWER8 based systems. Two models
(7042-CR7 and 7042-CR8) are available for ordering at the time of writing, but you can also
use one of the withdrawn models listed in Table 1-10.

Table 1-10 HMC models supporting POWER8 processor technology-based servers

Type-model Availability Description

7042-C08 Withdrawn IBM 7042 Model C08 Deskside Hardware Management Console

7042-CR5 Withdrawn IBM 7042 Model CR5 Rack-Mounted Hardware Management Console

7042-CR6 Withdrawn IBM 7042 Model CR6 Rack mounted Hardware Management Console

7042-CR7 Available IBM 7042 Model CR7 Rack mounted Hardware Management Console

7042-CR8 Available IBM 7042 Model CR8 Rack mounted Hardware Management Console

At the time of writing, base Licensed Machine Code Version 8 Revision 8.2.0 or later is
required to support the Power E870 (9119-MME) and Power E880 (9119-MHE).

Fix Central: You can download or order the latest HMC code from the Fix Central website:

http://www.ibm.com/support/fixcentral

Existing HMC models 7042 can be upgraded to Licensed Machine Code Version 8 to support
environments that might include POWER6®, POWER6+®,POWER7®, POWER7+ and
POWER8 processor-based servers.

If you want to support more than 254 partitions in total, the HMC will require a memory
upgrade to a minimum of 4 GB.

1.12 System racks

The Power E870 and E880 and its I/O drawers are designed to be mounted in the following
existing IBM racks: 7014-T00, 7014-T42, 7965-94Y, #0553, #0551, and #ER05.

However, for initial system orders, the racks must be ordered as machine type 7014-T42,
#0553. The 36U (1.8-meter) rack (#0551) and the 42U (2.0-meter) rack (#ER05) are available
for order only on Miscellaneous Equipment Specification (MES) upgrade orders only.

Shipping without a rack: If you require the system to be shipped without an IBM rack,
feature code #ER21 must be used to remove the IBM rack from the order. The server will
then ship as separate packages for installation into an existing rack.

The Power E870 and Power E880 use a new type of connector between system drawers.
Therefore, the systems do not require wider racks, and an OEM rack or cabinet which meet
the requirements can be used.

24 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch01.fm

Installing in non-IBM racks: The client is responsible for ensuring that the installation of
the drawer in the preferred rack or cabinet results in a configuration that is stable,
serviceable, safe, and compatible with the drawer requirements for power, cooling, cable
management, weight, and rail security.

1.12.1 IBM 7014 model T00 rack

The 1.8-meter (71-inch) model T00 is compatible with past and present IBM Power systems.
The features of the T00 rack are as follows:

򐂰 36U (EIA units) of usable space.
򐂰 Optional removable side panels.
򐂰 Optional highly perforated front door.
򐂰 Optional side-to-side mounting hardware for joining multiple racks.
򐂰 Standard business black or optional white color in OEM format.
򐂰 Increased power distribution and weight capacity.
򐂰 Supports both AC and DC configurations.
򐂰 The rack height is increased to 1926 mm (75.8 in.) if a power distribution panel is fixed to

the top of the rack.

򐂰 The #6068 feature provides a cost effective plain front door
򐂰 Weights are as follows:

– T00 base empty rack: 244 kg (535 lb.)
– T00 full rack: 816 kg (1795 lb.)
– Maximum Weight of Drawers is 572 kg (1260 lb.)
– Maximum Weight of Drawers in a zone 4 earthquake environment is 490 kg (1080 lb.).

This equates to 13.6 kg (30 lb.)/EIA.

1.12.2 IBM 7014 model T42 rack

The 2.0-meter (79.3-inch) Model T42 addresses the requirement for a tall enclosure to house
the maximum amount of equipment in the smallest possible floor space. The following
features differ in the model T42 rack from the model T00:

򐂰 The T42 rack has 42U (EIA units) of usable space (6U of additional space).
򐂰 The model T42 supports AC power only.
򐂰 Weights are as follows:

– T42 base empty rack: 261 kg (575 lb.)
– T42 full rack: 930 kg (2045 lb.)

򐂰 The feature #ERG7 provides an attractive black full-height rack door. The door is steel,

with a perforated flat front surface. The perforation pattern extends from the bottom to the
top of the door to enhance ventilation and provide some visibility into the rack.

򐂰 The feature #6069 provides a cost-effective plain front door.
򐂰 The feature #6249 provides a special acoustic door

Special door: The Power 780 logo rack door (#6250) is not supported.

Chapter 1. General description 25

5137ch01.fm Draft Document for Review October 14, 2014 10:19 am

1.12.3 IBM 7953 model 94Y rack

The 2.0-meter (79.3 inch) model 94Y rack has the following features:

򐂰 42U (EIA units)
򐂰 Weights:

– Base empty rack: 187 kg (221 lb.)
– Maximum load limit: 664 kg (1460 lb.)

The IBM 42U Slim Rack (7953-94Y) differs from the IBM 42U enterprise rack (7014-T42) in
several aspects. Both provide 42U of vertical space, are 1100 mm deep, and have an interior
rail-to-rail depth of 715 mm. However, the IBM 42U Slim Rack is 600 mm wide; the T42 is
645 mm wide with side covers. For clients with 2-foot floor tiles, the extra 45 mm (1.77-inch)
width of the enterprise rack can sometimes cause challenges when cutting holes in the floor
tiles for cabling.

The 42U Slim Rack has a lockable perforated front steel door, providing ventilation, physical
security, and visibility of indicator lights in the installed equipment within. In the rear, either a
lockable perforated rear steel door (#EC02) or a lockable rear door heat exchanger (#EC15)
is used. Lockable optional side panels (#EC03) increase the rack’s aesthetics, help control
airflow through the rack, and provide physical security. Multiple 42U Slim Racks can be bolted
together to create a rack suite (indicated with feature #EC04).

Weight calculation: Maximum weight limits must include everything that will be installed
in the rack. This must include servers, I/O drawers, PDUs, switches, and anything else
installed. In zone 4 earthquake environments the rack should be configured starting with
the heavier drawers at the bottom of the rack.

1.12.4 Feature code #0551 rack

The 1.8-meter rack (#0551) is a 36U (EIA units) rack. The rack that is delivered as #0551 is
the same rack delivered when you order the 7014-T00 rack. The included features might
differ. Several features that are delivered as part of the 7014-T00 must be ordered separately
with the #0551. The #0551 is not available for initial orders of Power E870 and E880 servers.

1.12.5 Feature code #0553 rack

The 2.0-meter rack (#0553) is a 42U (EIA units) rack. The rack that is delivered as #0553 is
the same rack delivered when you order the 7014-T42. The included features might differ.
Several features that are delivered as part of the 7014-T42 or must be ordered separately
with the #0553.

1.12.6 The AC power distribution unit and rack content

For rack models T00, T42 and the slim 94Y, 12-outlet PDUs are available. The PDUs
available include:

򐂰 PDUs Universal UTG0247 Connector (#7188)
򐂰 Intelligent PDU+ Universal UTG0247 Connector (#7109).

PDU mounting: Only horizontal PDUs are allowed in racks hosting Power E870 and
Power E880 system. Vertically mounted PDUs limit access to the cable routing space on
the side of the rack and cannot be used.

26 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch01.fm

41

42

39

40

37

38

35

36

33

34

31

32

29

30

27

28

25

26

23

24

21

22

19

20

17

18

15

16

13

14

11

12

09

10

07

08

05

06

03

04

01

02

41

42

39

40

37

38

35

36

33

34

31

32

29

30

27

28

25

26

23

24

21

22

19

20

17

18

15

16

13

14

11

12

09

10

07

08

05

06

03

04

01

02

System Node

#1

Control Unit

I/O Expansion Drawer

P4P3P1 P2

30A PDU

30A PDU

P1 P2

I/O Expansion Drawer

P1 P2

System Node

#2

Control Unit

I/O Expansion Drawer

I/O Expansion Drawer

P4P3P1 P2

P4P3P1 P2

30A PDU

30A PDU

30A PDU

30A PDU

P2P1

P2P1

When mounting the horizontal PDUs it is a good practice to place them almost at the top or
almost at the bottom of the rack leaving 2U or more of space at the very top or very bottom of
the open for cable management. Mounting a horizontal PDU in the middle of the rack is
generally not optimal for cable management.

For the Power E870 or E880 installed in IBM 7014 or FC 055x racks, the following PDU rules
apply:

򐂰 For PDU #7109 and #7188 when using 24 Amp power cord #6654, #6655, #6656, #6657,

or #6658, each pair of PDUs can power one Power E870 or Power E880 system nodes
and two I/O expansion drawers or eight I/O expansion drawers. 24A PDU cables are used
to supply 30A PDUs. In the Figure 1-12 on page 27 you can see the rack configuration
with two pairs of 30A PDUs suppling two system nodes configuration and two I/O
expansion drawers.

Figure 1-12 Two system node configuration and two I/O expansion drawers supplied by 30A PDUs

򐂰 For PDU #7109 and #7188 when using three phase power cords or 48 Amp power cords

#6491 or #6492, each pair of PDUs can power up to two Power E870 or Power E880
system nodes and two I/O expansion drawer or eight I/O expansion drawers. 48A PDU
cables are used to supply 60A PDU. In the Figure 1-13 on page 28 you can see the rack

Chapter 1. General description 27

5137ch01.fm Draft Document for Review October 14, 2014 10:19 am

41

42

39

40

37

38

35

36

33

34

31

32

29

30

27

28

25

26

23

24

21

22

19

20

17

18

15

16

13

14

11

12

09

10

07

08

05

06

03

04

01

02

41

42

39

40

37

38

35

36

33

34

31

32

29

30

27

28

25

26

23

24

21

22

19

20

17

18

15

16

13

14

11

12

09

10

07

08

05

06

03

04

01

02

System Node

#2

System Node

#1

Control Unit

I/O Expansion Drawer

I/O Expansion Drawer

I/O Expansion Drawer

I/O Expansion Drawer

EXP24S

EXP24S

EXP24S

EXP24S

P4P3P1 P2

P4P3P1 P2

60A PDU

60A PDU

60A PDU

60A PDU

P2P1

P2P1

P2P1

P2P1

P2P1

P2P1

P2P1

P2P1

configuration with two pairs of 60A PDUs suppling a two system node configuration, four
I/O expansion drawers, and four EXP24S disk drawers.

28 IBM Power Systems E870 and E880 Technical Overview and Introduction

Figure 1-13 Two system nodes configuration with I/O expansions and disk drawers supplied by 60A
PDUs

For detailed power cord requirements and power cord feature codes, see the IBM Power
Systems Hardware Information Center website:

<< add an infocenter link once is available>>

Power cord: Ensure that the appropriate power cord feature is configured to support the
power being supplied.

For rack integrated systems a minimum quantity of two PDUs (#7109 or #7188) are required.

The PDUs (#7109, #7188) support a wide range of country requirements and electrical power
specifications. The PDU receives power through a UTG0247 power line connector. Each PDU
requires one PDU-to-wall power cord. Various power cord features are available for countries
and applications by selecting a PDU-to-wall power cord, which must be ordered separately.
Each power cord provides the unique design characteristics for the specific power
requirements. To match new power requirements and save previous investments, these

Draft Document for Review October 14, 2014 10:19 am 5137ch01.fm

power cords can be requested with an initial order of the rack or with a later upgrade of the
rack features.

The PDUs have 12 client-usable IEC 320-C13 outlets. There are six groups of two outlets fed
by six circuit breakers. Each outlet is rated up to 10 Amps, but each group of two outlets is fed
from one 20 A circuit breaker.

The Universal PDUs are compatible with previous models.

Power cord and PDU: Based on the power cord that is used, the PDU can supply a range
of 4.8 - 21 kVA. The total kilovolt ampere (kVA) of all the drawers that are plugged into the
PDU must not exceed the power cord limitation.

Each system node mounted in the rack requires four power cords. For maximum
availability, be sure to connect power cords from the same system to two separate PDUs in
the rack, and to connect each PDU to independent power sources.

Chapter 1. General description 29

5137ch01.fm Draft Document for Review October 14, 2014 10:19 am

30 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch02.fm

2

Chapter 2. Architecture and technical

overview

This chapter describes the overall system architecture for the IBM Power System E870
(9119-MME) and IBM Power System E880 (9119-MHE) servers. The bandwidths that are
provided throughout the section are theoretical maximums that are used for reference.

The speeds that are shown are at an individual component level. Multiple components and
application implementation are key to achieving the best performance.

Always do the performance sizing at the application workload environment level and evaluate
performance by using real-world performance measurements and production workloads.

© Copyright IBM Corp. 2014. All rights reserved. 31

5137ch02.fm Draft Document for Review October 14, 2014 10:19 am

SCU Interface

P8 (1)

Memory

Controller 0

Memory

Controller 1

PHB 0

PHB 1

P8 (2)

PHB 0

PHB 1

P8 (0)

PHB 0

PHB 1

P8 (3)

PHB 0

PHB 1

SMP

X Bus

SMP

X Bus

SMP

X Bus

SMP

X Bus

SMP

A Bus

SMP

A Bus

SMP

A Bus

SMP

A Bus

PCIe Gen3 x16 slot

PCIe Gen3 x16 slot

PCIe Gen3 x16 slot

PCIe Gen3 x16 slot

PCIe Gen3 x16 slot

PCIe Gen3 x16 slot

PCIe Gen3 x16 slot

PCIe Gen3 x16 slot

CDIMM
CDIMM
CDIMM
CDIMM

CDIMM
CDIMM
CDIMM
CDIMM

Memory

Controller 0

Memory

Controller 1

CDIMM
CDIMM
CDIMM
CDIMM

CDIMM
CDIMM
CDIMM
CDIMM

Memory

Controller 0

Memory

Controller 1

CDIMM
CDIMM
CDIMM
CDIMM

CDIMM
CDIMM
CDIMM
CDIMM

Memory

Controller 0

Memory

Controller 1

CDIMM
CDIMM
CDIMM
CDIMM

CDIMM
CDIMM
CDIMM
CDIMM

To other

enclosures

To other

enclosures

To other

enclosures

To other

enclosures

SCU Interface

card

12v power

to SCU

Service

Processor

Connection

Clock

Interface

To SCU

Clock

SCU Interface

SCU Interface

card

12v power

to SCU

Service

Processor

Connection

Clock

Interface

To SCU

Clock

25.6 GB/s

76.8 GB/s

15.75 GB/s

28.8 GB/s

2.1 Logical diagrams

This section contains logical diagrams for the Power E870 and E880.

Figure 2-1 shows the logical system diagram for a single system node of a Power E870 or
Power E880.

Figure 2-1 Logical system diagram for a system node of a Power E870 or a Power E880

32 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch02.fm

Power in

Operator Panel

From

CEC

Power in

From

CEC

Power in

From

CEC

Power in

From

CEC

USB

cable

to

PCIe card

DVD

FSP 0 FSP 1Clock 0 Clock 1

VPD

Figure 2-2 on page 33 shows the logical system diagram for the system control unit of a
Power E870 or a Power E880.

Figure 2-2 Logical system diagram for the system control unit

Chapter 2. Architecture and technical overview 33

5137ch02.fm Draft Document for Review October 14, 2014 10:19 am

Drawer 1

Drawer 0

X3

X1

X0

X3

X1

X0

X3

X1

X0

X3

X1

X0

X0

X1

X3

X0

X1

X3

X0

X1

X3

X0

X1

X3

P0

P2

P3

P1

T1 T2 T5 T6T3 T4

T6

T5

T4

T3

T2

T1

A2

A1

A0

A2

A1

A0

A2

A1

A0

A2

A1

A0

A0

A1

A2

A0

A1

A2

A0

A1

A2

A0

A1

A2

P1

P3

P2

P0

Flexible symmetric multiprocessing (SMP) cables are used to connect system nodes when a
Power E870 or Power E880 is configured with more than one system node. Figure 2-3 on
page 34 shows the SMP connection topology for a two drawer Power E870 or Power E880
system.

Figure 2-3 SMP connection topology for a two-drawer Power E870 or Power E880

34 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch02.fm

T1 T2 T5 T6T3 T4

P3

X3

X1

X0

A2

A1

A0

Drawer 3

P2

X3

X1

X0

A2

A1

A0

P0

X3

X1

X0

A2

A1

A0

P1

X3

X1

X0

A2

A1

A0

Drawer 2

T1 T2 T5 T6T3 T4

Drawer 1

X3

X1

X0

X3

X1

X0

X3

X1

X0

X3

X1

X0

P0

P2

P3

P1

A2

A1

A0

A2

A1

A0

A2

A1

A0

A2

A1

A0

T6

T5

T4

T3

T2

T1

A0

A1

A2

A0

A1

A2

A0

A1

A2

A0

A1

A2

X0

X1

X3

X0

X1

X3

X0

X1

X3

X0

X1

X3

P1

P3

P2

P0

Figure 2-4 shows the SMP connection topology for a three-drawer Power E880.

Figure 2-4 SMP connection topology for a three-drawer Power E880

Chapter 2. Architecture and technical overview 35

5137ch02.fm Draft Document for Review October 14, 2014 10:19 am

T1 T2 T5T3 T4

P3

X3

X1

X0

A2

A1

A0

Drawer 3

P2

X3

X1

X0

A2

A1

A0

P0

X3

X1

X0

A2

A1

A0

P1

X3

X1

X0

A2

A1

A0

Drawer 2

Drawer 4

T1 T2 T5 T6T3 T4

Drawer 1

X3

X1

X0

P0

P2

P3

P1

X3

X1

X0

X3

X1

X0

X3

X1

X0

A2

A1

A0

A2

A1

A0

A2

A1

A0

A2

A1

A0

T6

T6

T5

T2

T4 T3

T1

T6

T5

T4

T3

T2

T1

A2

A1

A0

A2

A1

A0

A2

A1

A0

A2

A1

A0

X3

X1

X0

X3

X1

X0

X3

X1

X0

X3

X1

X0

P0

P2

P3

P1

A0

A1

A2

A0

A1

A2

A0

A1

A2

A0

A1

A2

X0

X1

X3

X0

X1

X3

X0

X1

X3

X0

X1

X3

P1

P3

P2

P0

Figure 2-5 shows the SMP connection topology for a four-drawer Power E880.

Figure 2-5 SMP connection topology for a four-drawer Power E880

2.2 The IBM POWER8 processor

This section introduces the latest processor in the IBM Power Systems product family, and
describes its main characteristics and features.

36 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch02.fm

2.2.1 POWER8 processor overview

The POWER8 processor is manufactured using the IBM 22 nm Silicon-On-Insulator (SOI)
technology. Each chip is 649 mm

2

and contains 4.2 billion transistors. As shown in Figure 2-6,
the chip contains 12 cores, two memory controllers, PCIe Gen3 I/O controllers, and an
interconnection system that connects all components within the chip. On some systems only
6, 8, 10, or 12 cores per processor may be available to the server. Each core has 512 KB of
L2 cache, and all cores share 96 MB of L3 embedded DRAM (eDRAM). The interconnect
also extends through module and board technology to other POWER8 processors in addition
to DDR3 memory and various I/O devices.

POWER8 systems use memory buffer chips to interface between the POWER8 processor
and DDR3 or DDR4 memory

1

. Each buffer chip also includes an L4 cache to reduce the

latency of local memory accesses.

Figure 2-6 The POWER8 processor chip

The POWER8 processor is for system offerings from single-socket servers to multi-socket
Enterprise servers. It incorporates a triple-scope broadcast coherence protocol over local and
global SMP links to provide superior scaling attributes. Multiple-scope coherence protocols
reduce the amount of SMP link bandwidth that is required by attempting operations on a
limited scope (single chip or multi-chip group) when possible. If the operation cannot
complete coherently, the operation is reissued using a larger scope to complete the operation.

1

At the time of the publication, the available POWER8 processor-based systems use DDR3 memory.

Chapter 2. Architecture and technical overview 37

5137ch02.fm Draft Document for Review October 14, 2014 10:19 am

The following are additional features that can augment the performance of the POWER8
processor:

򐂰 Support for DDR3 and DDR4 memory through memory buffer chips that offload the

memory support from the POWER8 memory controller.

򐂰 Each memory CDIMM has 16 MB of L4 cache within the memory buffer chip that reduces

the memory latency for local access to memory behind the buffer chip; the operation of the
L4 cache is not apparent to applications running on the POWER8 processor. Up to 128
MB of L4 cache can be available for each POWER8 processor.

򐂰 Hardware transactional memory.
򐂰 On-chip accelerators, including on-chip encryption, compression, and random number

generation accelerators.

򐂰 Coherent Accelerator Processor Interface, which allows accelerators plugged into a PCIe

slot to access the processor bus using a low latency, high-speed protocol interface.

򐂰 Adaptive power management.

There are two versions of the POWER8 processor chip. Both chips use the same building
blocks. The scale-out systems use a 6-core version of POWER8. The 6-core chip is installed
in pairs in a Dual Chip Module (DCM) that plugs into a socket in the system board of the
systems. Functionally, it works as a single chip module (SCM).

Figure 2-7 shows a graphic representation of the 6-core processor. A 6-core processor is only
available on the scale-out systems. It is shown here for informational purposes.

Figure 2-7 6-core POWER8 processor chip

38 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch02.fm

Table 2-1 summarizes the technology characteristics of the POWER8 processor.

Table 2-1 Summary of POWER8 processor technology

Technology POWER8 processor

Die size 649 mm

Fabrication technology 򐂰 22 nm lithography

򐂰 Copper interconnect
򐂰 SOI
򐂰 eDRAM

Maximum processor cores 6 or 12

Maximum execution threads core/chip 8/96

Maximum L2 cache core/chip 512 KB/6 MB

Maximum On-chip L3 cache core/chip 8 MB/96 MB

Maximum L4 cache per chip 128 MB

Maximum memory controllers 2

SMP design-point 16 sockets with IBM POWER8 processors

2

Compatibility With prior generations of POWER® processor

2.2.2 POWER8 processor core

The POWER8 processor core is a 64-bit implementation of the IBM Power Instruction Set
Architecture (ISA) Version 2.07 and has the following features:

򐂰 Multi-threaded design, which is capable of up to eight-way simultaneous multithreading

(SMT)

򐂰 32 KB, eight-way set-associative L1 instruction cache
򐂰 64 KB, eight-way set-associative L1 data cache
򐂰 Enhanced prefetch, with instruction speculation awareness and data prefetch depth

awareness

򐂰 Enhanced branch prediction, using both local and global prediction tables with a selector

table to choose the best predictor

򐂰 Improved out-of-order execution
򐂰 Two symmetric fixed-point execution units
򐂰 Two symmetric load/store units and two load units, all four of which can also run simple

fixed-point instructions

򐂰 An integrated, multi-pipeline vector-scalar floating point unit for running both scalar and

SIMD-type instructions, including the Vector Multimedia eXtension (VMX) instruction set
and the improved Vector Scalar eXtension (VSX) instruction set, and capable of up to
eight floating point operations per cycle (four double precision or eight single precision)

򐂰 In-core Advanced Encryption Standard (AES) encryption capability
򐂰 Hardware data prefetching with 16 independent data streams and software control
򐂰 Hardware decimal floating point (DFP) capability

Chapter 2. Architecture and technical overview 39

5137ch02.fm Draft Document for Review October 14, 2014 10:19 am

More information about Power ISA Version 2.07 can be found at the following website:

https://www.power.org/documentation/power-isa-version-2-07/

Figure 2-8 shows a picture of the POWER8 core, with some of the functional units
highlighted.

Figure 2-8 POWER8 processor core

2.2.3 Simultaneous multithreading

POWER8 processor advancements in multi-core and multi-thread scaling are remarkable. A
significant performance opportunity comes from parallelizing workloads to enable the full
potential of the microprocessor, and the large memory bandwidth. Application scaling is
influenced by both multi-core and multi-thread technology.

Simultaneous Multithreading (SMT) allows a single physical processor core to simultaneously
dispatch instructions from more than one hardware thread context. With SMT, each POWER8
core can present eight hardware threads. Because there are multiple hardware threads per
physical processor core, additional instructions can run at the same time. SMT is primarily
beneficial in commercial environments where the speed of an individual transaction is not as
critical as the total number of transactions that are performed. SMT typically increases the
throughput of workloads with large or frequently changing working sets, such as database
servers and web servers.

Table 2-2 on page 41 shows a comparison between the different POWER processors in terms
of SMT capabilities that are supported by each processor architecture.

40 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch02.fm

Table 2-2 SMT levels that are supported by POWER processors

Technology Cores/system Maximum SMT mode Maximum hardware

threads per system

IBM POWER4 32 Single Thread (ST) 32

IBM POWER5 64 SMT2 128

IBM POWER6 64 SMT2 128

IBM POWER7 256 SMT4 1024

IBM POWER8 192 SMT8 1536

The architecture of the POWER8 processor, with its larger caches, larger cache bandwidth,
and faster memory, allows threads to have faster access to memory resources, which
translates in to a more efficient usage of threads. Because of that, POWER8 allows more
threads per core to run concurrently, increasing the total throughput of the processor and of
the system.

2.2.4 Memory access

On the Power E870 and Power E880, each POWER8 processor has two memory controllers,
each connected to four memory channels. Each memory channel operates at 1600 MHz and
connects to a DIMM. Each DIMM on a POWER8 system has a memory buffer that is
responsible for many functions that were previously on the memory controller, such as
scheduling logic and energy management. The memory buffer also has 16 MB of level 4 (L4)
cache.

On the Power E870 each memory channel can address up to 64 GB. Therefore a single
system node can address 2 TB of memory. A two node system can address up to 4 TB of
memory.

On the Power E880 each memory channel can address up to 128 GB. Therefore a single
system node can address 4 TB of memory. A two node system can address up to 8 TB of
memory and a four node system can address up to 16 TB of memory.

Chapter 2. Architecture and technical overview 41

5137ch02.fm Draft Document for Review October 14, 2014 10:19 am

POWER8

Processor

Memory
Buffers

DRAM
Chips

Memory
Buffers

DRAM

Chips

Figure 2-9 gives a simple overview of the POWER8 processor memory access structure in
the Power E870 and Power E880.

Figure 2-9 Overview of POWER8 memory access structure

2.2.5 On-chip L3 cache innovation and Intelligent Cache

Similar to POWER7 and POWER7+, the POWER8 processor uses a breakthrough in material
engineering and microprocessor fabrication to implement the L3 cache in eDRAM and place it
on the processor die. L3 cache is critical to a balanced design, as is the ability to provide good
signaling between the L3 cache and other elements of the hierarchy, such as the L2 cache or
SMP interconnect.

The on-chip L3 cache is organized into separate areas with differing latency characteristics.
Each processor core is associated with a fast 8 MB local region of L3 cache (FLR-L3) but also
has access to other L3 cache regions as shared L3 cache. Additionally, each core can
negotiate to use the FLR-L3 cache that is associated with another core, depending on the
reference patterns. Data can also be cloned and stored in more than one core’s FLR-L3
cache, again depending on the reference patterns. This
enables the POWER8 processor to optimize the access to L3 cache lines and minimize
overall cache latencies.

Figure 2-6 on page 37 and Figure 2-7 on page 38 show the on-chip L3 cache, and highlight
one fast 8 MB L3 region closest to a processor core.

The benefits of using eDRAM on the POWER8 processor die is significant for several
reasons:

򐂰 Latency improvement

򐂰 Bandwidth improvement

A six-to-one latency improvement occurs by moving the L3 cache on-chip compared to L3
accesses on an external (on-ceramic) ASIC.

Intelligent Cache management

A 2x bandwidth improvement occurs with on-chip interconnect. Frequency and bus sizes
are increased to and from each core.

42 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch02.fm

򐂰 No off-chip driver or receivers

Removing drivers or receivers from the L3 access path lowers interface requirements,
conserves energy, and lowers latency.

򐂰 Small physical footprint

The performance of eDRAM when implemented on-chip is similar to conventional SRAM
but requires far less physical space. IBM on-chip eDRAM uses only a third of the
components that conventional SRAM uses, which has a minimum of six transistors to
implement a 1-bit memory cell.

򐂰 Low energy consumption

The on-chip eDRAM uses only 20% of the standby power of SRAM.

2.2.6 Level 4 cache and memory buffer

POWER8 processor-based systems introduce an additional level of memory hierarchy. The
Level 4 (L4) cache is implemented together with the memory buffer in the Custom DIMM
(CDIMM). Each memory buffer contains 16 MB of L4 cache. Figure 2-10 shows a picture of
the memory buffer, where you can see the 16 MB L4 cache, and processor links and memory
interfaces.

Figure 2-10 Memory buffer chip

Table 2-3 shows a comparison of the different levels of cache in the POWER7, POWER7+,
and POWER8 processors.

Table 2-3 POWER8 cache hierarchy

Cache POWER7 POWER7+ POWER8

L1 instruction cache:
Capacity/associativity

L1 data cache:
Capacity/associativity
bandwidth

32 KB, 4-way 32 KB, 4-way 32 KB, 8-way

32 KB, 8-way
Two 16 B reads or
one 16 B writes per cycle

32 KB, 8-way
Two 16 B reads or
one 16 B writes per cycle

Chapter 2. Architecture and technical overview 43

64 KB, 8-way
Two 16 B reads or
one 16 B writes per cycle

5137ch02.fm Draft Document for Review October 14, 2014 10:19 am

Cache POWER7 POWER7+ POWER8

L2 cache:
Capacity/associativity
bandwidth

L3 cache:
Capacity/associativity
bandwidth

L4 cache:
Capacity/associativity
bandwidth

256 KB, 8-way
Private
32 B reads and 16 B writes
per cycle

On-Chip
4 MB/core, 8-way
16 B reads and 16 B writes
per cycle

N/A N/A On-Chip

For more information about the POWER8 memory subsystem, see 2.3, “Memory subsystem”
on page 47.

2.2.7 Hardware transactional memory

Transactional memory is an alternative to lock-based synchronization. It attempts to simplify
parallel programming by grouping read and write operations and running them like a single
operation. Transactional memory is like database transactions where all shared memory
accesses and their effects are either committed all together or discarded as a group. All
threads can enter the critical region simultaneously. If there are conflicts in accessing the
shared memory data, threads try accessing the shared memory data again or are stopped
without updating the shared memory data. Therefore, transactional memory is also called a
lock-free synchronization. Transactional memory can be a competitive alternative to
lock-based synchronization.

256 KB, 8-way
Private
32 B reads and 16 B writes
per cycle

On-Chip
10 MB/core, 8-way
16 B reads and 16 B writes
per cycle

512 KB, 8-way
Private
32 B reads and 16 B writes
per cycle

On-Chip
8 MB/core, 8-way
32 B reads and 32 B writes
per cycle

16 MB/buffer chip, 16-way
Up to 8 buffer chips per
socket

Transactional Memory provides a programming model that makes parallel programming
easier. A programmer delimits regions of code that access shared data and the hardware
runs these regions atomically and in isolation, buffering the results of individual instructions,
and retrying execution if isolation is violated. Generally, transactional memory allows
programs to use a programming style that is close to coarse-grained locking to achieve
performance that is close to fine-grained locking.

Most implementations of transactional memory are based on software. The POWER8
processor-based systems provide a hardware-based implementation of transactional memory
that is more efficient than the software implementations and requires no interaction with the
processor core, therefore allowing the system to operate at maximum performance.

2.2.8 Coherent Accelerator Processor Interface

The Coherent Accelerator Interface Architecture (CAIA) defines a coherent accelerator
interface structure for attaching peripherals to Power Systems.

The Coherent Accelerator Processor Interface (CAPI) can attach accelerators that have
coherent shared memory access to the processors in the server and share full virtual address
translation with these processors, using a standard PCIe Gen3 bus.

Applications can have customized functions in Field Programmable Gate Arrays (FPGA) and
be able to enqueue work requests directly in shared memory queues to the FPGA, and using
the same effective addresses (pointers) it uses for any of its threads running on a host

44 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch02.fm

Custom

Hardware

Application

CAPP

Coh erenc e Bus

PSL

FPGA or ASIC

POW ER8

PCIe G en3

Tra nsport f o r en ca psul ated messag es

processor. From the practical perspective, CAPI allows a specialized hardware accelerator to
be seen as an additional processor in the system, with access to the main system memory,
and coherent communication with other processors in the system.

The benefits of using CAPI include the ability to access shared memory blocks directly from
the accelerator, perform memory transfers directly between the accelerator and processor
cache, and reduction in the code path length between the adapter and the processors. This is
because the adapter is not operating as a traditional I/O device, and there is no device driver
layer to perform processing. It also presents a simpler programming model.

Figure 2-11 shows a high-level view of how an accelerator communicates with the POWER8
processor through CAPI. The POWER8 processor provides a Coherent Attached Processor
Proxy (CAPP), which is responsible for extending the coherence in the processor
communications to an external device. The coherency protocol is tunneled over standard
PCIe Gen3, effectively making the accelerator part of the coherency domain.

The accelerator adapter implements the Power Service Layer (PSL), which provides address
translation and system memory cache for the accelerator functions. The custom processors
on the board, consisting of an FPGA or an Application Specific Integrated Circuit (ASIC) use
this layer to access shared memory regions, and cache areas as through they were a
processor in the system. This ability greatly enhances the performance of the data access for
the device and simplifies the programming effort to use the device. Instead of treating the
hardware accelerator as an I/O device, it is treated as a processor. This eliminates the
requirement of a device driver to perform communication, and the need for Direct Memory
Access that requires system calls to the operating system kernel. By removing these layers,
the data transfer operation requires fewer clock cycles in the processor, greatly improving the
I/O performance.

Figure 2-11 CAPI accelerator that is attached to the POWER8 processor

The implementation of CAPI on the POWER8 processor allows hardware companies to
develop solutions for specific application demands and use the performance of the POWER8
processor for general applications and custom acceleration of specific functions using a
hardware accelerator, with a simplified programming model and efficient communication with
the processor and memory resources.

Chapter 2. Architecture and technical overview 45

5137ch02.fm Draft Document for Review October 14, 2014 10:19 am

2.2.9 Power management and system performance

The POWER8 processor has power saving and performance enhancing features that can be
used to lower overall energy usage, while yielding higher performance when needed. The
following modes can be enabled and modified to use these features.

Dynamic Power Saver: Favor Performance

This mode is intended to provide the best performance. If the processor is being used even
moderately, the frequency is raised to the maximum frequency possible to provide the best
performance. If the processors are lightly used, the frequency is lowered to the minimum
frequency, which is potentially far below the nominal shipped frequency, to save energy. The
top frequency that is achieved is based on system type and is affected by environmental
conditions. Also, when running at the maximum frequency, more energy is being consumed,
which means this mode can potentially cause an increase in overall energy consumption.

Dynamic Power Saver: Favor Power

This mode is intended to provide the best performance per watt consumed. The processor
frequency is adjusted based on the processor usage to maintain the workload throughput
without using more energy than required to do so. At high processor usage levels, the
frequency is raised above nominal, as in the Favor Performance mode. Likewise, at low
processor usage levels, the frequency is lowered to the minimum frequency. The frequency
ranges are the same for the two Dynamic Power Saver modes, but the algorithm that
determines which frequency to set is different.

Dynamic Power Saver: Tunable Parameters

The Dynamic Power Saver: Favor Performance and Dynamic Power Saver: Favor Power
modes are tuned to provide both energy savings and performance increases. However, there
might be situations where only top performance is of concern, or, conversely, where peak
power consumption is an issue. The tunable parameters can be used to modify the setting of
the processor frequency in these modes to meet these various objectives. Modifying these
parameters should be done only by advanced users. If you must address any issues
concerning the Tunable Parameters, IBM support personal should be directly involved in the
parameter value selection.

Idle Power Saver

This mode is intended to save the maximum amount of energy when the system is nearly idle.
When the processors are found to be nearly idle, the frequency of all processors is lowered to
the minimum. Additionally, workloads are dispatched onto a smaller number of processor
cores so that the other processor cores can be put into a low energy usage state. When
processor usage increases, the process is reversed: The processor frequency is raised back
up to nominal, and the workloads are spread out once again over all of the processor cores.
There is no performance boosting aspect in this mode, but entering or exiting this mode might
affect overall performance. The delay times and usage levels for entering and exiting this
mode can be adjusted to allow for more or less aggressive energy savings.

The controls for all of these modes are available on the Advanced System Management
Interface (ASMI) and are described in more detail in a white paper that is found at the
following link:

http://public.dhe.ibm.com/common/ssi/ecm/en/pow03039usen/POW03039USEN.PDF

For more information, see 2.13, “Energy management” on page 105.

46 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch02.fm

2.2.10 Comparison of the POWER7, POWER7+, and POWER8 processors

Table 2-4 shows comparable characteristics between the generations of POWER7,
POWER7+, and POWER8 processors.

Table 2-4 Comparison of technologies for the POWER8 processor and the prior generations

Characteristics POWER7 POWER7+ POWER8

Technology 45 nm 32 nm 22 nm

Die size 567 mm

Number of transistors 1.2 billion 2.1 billion 4.2 billion

Maximum cores 8 8 12

2

567 mm

2

649 mm

2

Maximum SMT
threads per core

Maximum frequency 4.25 GHz 4.4 GHz 4.35 GHz

L2 Cache 256 KB per core 256 KB per core 512 KB per core

L3 Cache 4 MB or 8 MB of

Memory support DDR3 DDR3 DDR3 and DDR4

I/O busGX++GX++PCIe Gen3

2.3 Memory subsystem

The Power E870 can have up to two system nodes per system with each system node having
32 DDR3 CDIMM slots capable of supporting 16 GB, 32 GB, and 64 GB CDIMMs running at
speeds of 1600 MHz. This allows for a maximum system memory of 4 TB for the
64 DDR3 CDIMMs slots in a system comprised of two system nodes.

4 threads 4 threads 8 threads

FLR-L3 cache per core
with each core having
access to the full 32
MB of L3 cache,
on-chip eDRAM

10 MB of FLR-L3
cache per core with
each core having
access to the full 80
MB of L3 cache,
on-chip eDRAM

8 MB of FLR-L3 cache
per core with each
core having access to
the full 96 MB of L3
cache, on-chip
eDRAM

The Power E880 can have up to four system nodes per system with each system node having
32 DDR3 CDIMM slots capable of supporting 16 GB, 32 GB, 64 GB, and 128 GB CDIMMs
running at speeds of 1600 MHz. This allows for a maximum system memory of 16 TB for the
128 DDR3 CDIMMs slots of a system comprised of four system nodes.

Note: At the time of writing, the Power E880 supports only two system nodes per server.
This allows for a maximum of 8 TB of memory for the 64 DDR3 CDIMM slots.

The memory on the systems are Capacity Upgrade on Demand capable, allowing for the
purchase of additional memory capacity and dynamically activate it when needed. It is
required that at least 50% of the installed RAM capacity is active.

The Power E870 and E880 servers support an optional feature called Active Memory
Expansion. This allows the effective maximum memory capacity to be much larger than the
true physical memory. This feature runs innovative compression and decompression of
memory content by using a dedicated coprocessor present on each POWER8 processor to

Chapter 2. Architecture and technical overview 47

5137ch02.fm Draft Document for Review October 14, 2014 10:19 am

provide memory expansion up to 125%, depending on the workload type and its memory
usage. As an example, a server with 256 GB of RAM physically installed can effectively be
expanded over 512 GB of RAM. This approach can enhance virtualization and server
consolidation by allowing a partition to do more work with the same physical amount of
memory or allowing a server to run more partitions and do more work with the same physical
amount of memory.

2.3.1 Custom DIMM

Custom DIMMs (CDIMMs) are innovative memory DIMMs that house industry-standard
DRAM memory chips and a set of components that allow for higher bandwidth, lower latency
communications and increased availability. These components include:

򐂰 Memory Scheduler
򐂰 Memory Management (RAS Decisions & Energy Management)
򐂰 Memory Buffer

By adopting this architecture for the memory DIMMs, several decisions and processes
regarding memory optimizations are run internally into the CDIMM. This saves bandwidth and
allows for faster processor-to-memory communications. This also allows for a more robust
RAS. For more information, see Chapter 4, “Reliability, availability, and serviceability” on
page 145.

The CDIMMs exists in two different form factors, a 152 SDRAM design named the Tall
CDIMM and an 80 SDRAM design named the Short CDIMM. Each design will be comprised
of multiple 4 GB SDRAM devices depending on its total capacity. The CDIMMs slots for the
Power E870 and Power E880 are tall CDIMMs slots. A filler is added to the short CDIMM
allowing it to properly latch into the same physical location of a tall CDIMM and allows for
proper airflow and ease of handling. Tall CDIMMs slots allow for larger DIMM sizes and the
adoption of future technologies more seamlessly.

48 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch02.fm

A detailed diagram of the CDIMMs available for the Power E870 and Power E880 can be
seen in Figure 2-12.

Figure 2-12 Short CDIMM and Tall CDIMM details

The Memory Buffer is a L4 cache and is built on eDRAM technology (same as the L3 cache),
which has a lower latency than regular SRAM. Each CDIMM has 16 MB of L4 cache and a
fully populated Power E870 server has 1 GB of L4 Cache while a fully populated Power E880
has 2 GB of L4 Cache. The L4 Cache performs several functions that have direct impact on
performance and bring a series of benefits for the Power E870 and Power E880:

򐂰 Reduces energy consumption by reducing the number of memory requests.
򐂰 Increases memory write performance by acting as a cache and by grouping several

random writes into larger transactions.

򐂰 Partial write operations that target the same cache block are gathered within the L4 cache

before being written to memory, becoming a single write operation.

򐂰 Reduces latency on memory access. Memory access for cached blocks has up to 55%

lower latency than non-cached blocks.

2.3.2 Memory placement rules

For the Power E870 and Power E880, each memory feature code provides four CDIMMs.
Therefore a maximum of 8 memory feature codes per system node are allowed in order to fill
all the 32 DDR3 CDIMM slots.

All the memory CDIMMs are capable of capacity upgrade on demand and must have a
minimum of 50% of its physical capacity activated. For example, the minimum installed

Chapter 2. Architecture and technical overview 49

5137ch02.fm Draft Document for Review October 14, 2014 10:19 am

memory for both servers is 256 GB of RAM, whereas they can have a minimum of 128 GB of
RAM active.

For the Power E870 the following memory options are orderable:

򐂰 64 GB (4 X 16 GB) CDIMMs, 1600 MHz DDR3 DRAM (#EM8J)
򐂰 128 GB (4 X 32 GB) CDIMMs, 1600 MHz DDR3 DRAM (#EM8K)
򐂰 256 GB (4 X 64 GB) CDIMMs, 1600 MHz DDR3 DRAM (#EM8L)

For the Power E880 the following memory options are orderable:

򐂰 64 GB (4 X 16 GB) CDIMMs, 1600 MHz DDR3 DRAM (#EM8J)
򐂰 128 GB (4 X 32 GB) CDIMMs, 1600 MHz DDR3 DRAM (#EM8K)
򐂰 256 GB (4 X 64 GB) CDIMMs, 1600 MHz DDR3 DRAM (#EM8L)
򐂰 512 GB (4 X 128 GB) CDIMMs, 1600 MHz DDR3 DRAM (#EM8M)

Each processor has two memory controllers. These memory controllers must have at least a
pair of CDIMMs attached to it. This set of mandatory four CDIMMs is called memory quad. A
logical diagram of a POWER8 processor with its two memory quads can be seen on
Figure 2-13.

Figure 2-13 Logical diagram of a POWER8 processor and its two memory quads

The basic rules for memory placement follows:

򐂰 Each feature code equals a set of four physical CDIMMs; a memory quad.
򐂰 Each installed processor must have at least one memory quad populated which equals to

at least one feature code per installed processor.

򐂰 A given processor can only have four or eight CDIMMs attached to it.
򐂰 All the CDIMMs connected to the same POWER8 processor must be identical. However, it

is permitted to mix different CDIMM sizes between different POWER8 processors on a
system.

򐂰 At least 50% of the installed memory must be activated via memory activation features.

The suggested approach is to install memory evenly across all processors in the system and
across all system nodes in a system. Balancing memory across the installed processors
allows memory access in a consistent manner and typically results in the best possible
performance for your configuration. You should account for any plans for future memory
upgrades when you decide which memory feature size to use at the time of the initial system
order.

50 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch02.fm

POWER8
Processor

1

MC

1

MC

0

POWER8
Processor

3

MC

1

MC

0

POWER8
Processor

2

MC

1

MC

0

POWER8
Processor

0

MC

1

MC

0

P1-C52 (Quad 1)

P1-C51 (Quad 1)

P1-C50 (Quad 5)

P1-C49 (Quad 5)

P1-C48 (Quad 5)

P1-C47 (Quad 5)

P1-C46 (Quad 1)

P1-C45 (Quad 1)

P1-C44 (Quad 2)

P1-C43 (Quad 2)

P1-C42 (Quad 6)

P1-C41 (Quad 6)

P1-C40 (Quad 6)

P1-C39 (Quad 6)

P1-C38 (Quad 2)

P1-C37 (Quad 2)

P1-C36 (Quad 3)

P1-C35 (Quad 3)

P1-C34 (Quad 7)

P1-C33 (Quad 7)

P1-C32 (Quad 7)

P1-C31 (Quad 7)

P1-C30 (Quad 3)

P1-C29 (Quad 3)

P1-C28 (Quad 4)

P1-C27 (Quad 4)

P1-C26 (Quad 8)

P1-C25 (Quad 8)

P1-C24 (Quad 8)

P1-C23 (Quad 8)

P1-C22 (Quad 4)

P1-C21 (Quad 4)

Location (Memory Quad)

Front

Rear

A physical diagram with the location codes of the memory CDIMMs of a system node, as well
as their grouping as memory quads can be seen on Figure 2-14 on page 51.

Figure 2-14 System node physical diagram with location codes for CDIMMs

Each system node has eight memory quads identified by the different colors on Figure 2-14.
The location codes for the slots on each memory quad are:

򐂰 Quad 1: P1-C45, P1-C46, P1-C51, and P1-C52
򐂰 Quad 2: P1-C37, P1-C38, P1-C43, and P1-C44
򐂰 Quad 3: P1-C29, P1-C30, P1-C35, and P1-C36
򐂰 Quad 4: P1-C21, P1-C22, P1-C27, and P1-C28
򐂰 Quad 5: P1-C47, P1-C48, P1-C49, and P1-C50
򐂰 Quad 6: P1-C39, P1-C40, P1-C41, and P1-C42
򐂰 Quad 7: P1-C31, P1-C32, P1-C33, and P1-C34
򐂰 Quad 8: P1-C23, P1-C24, P1-C25, and P1-C26

Chapter 2. Architecture and technical overview 51

5137ch02.fm Draft Document for Review October 14, 2014 10:19 am

Table 2-5 shows the CDIMM plugging order for a Power E870 or E880 with a single system
node:

Table 2-5 Optimal CDIMM memory quad placement for a single system node

System Node 1

Processor 0 Processor 2 Processor 3 Processor 1

Quad 1 Quad 5 Quad 2 Quad 6 Quad 3 Quad 7 Quad 4 Quad 8

15 2637 4 8

Notes:

Memory quads 1-4 must be populated.
Memory quads on the same processor must be populated with CDIMMs of the same capacity.

Table 2-6 shows the CDIMM plugging order for a Power E870 or E880 with two system
nodes:

Table 2-6 Optimal CDIMM memory quad placement for two system nodes

System Node 1

Processor 0 Processor 2 Processor 3 Processor 1

Quad 1 Quad 5 Quad 2 Quad 6 Quad 3 Quad 7 Quad 4 Quad 8

19 2113134 15

System Node 2

Processor 0 Processor 2 Processor 3 Processor 1

Quad 1 Quad 5 Quad 2 Quad 6 Quad 3 Quad 7 Quad 4 Quad 8

5106127148 16

Notes:

Memory quads 1-8 must be populated.
Memory quads on the same processor must be populated with CDIMMs of the same capacity.

52 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch02.fm

Table 2-7 shows the CDIMM plugging order for a Power E880 with a three system nodes:

Table 2-7 Optimal CDIMM memory quad placement for three system nodes

System Node 1

Processor 0 Processor 2 Processor 3 Processor 1

Quad 1 Quad 5 Quad 2 Quad 6 Quad 3 Quad 7 Quad 4 Quad 8

1132163194 22

System Node 2

Processor 0 Processor 2 Processor 3 Processor 1

Quad 1 Quad 5 Quad 2 Quad 6 Quad 3 Quad 7 Quad 4 Quad 8

5146177208 23

System Node 3

Processor 0 Processor 2 Processor 3 Processor 1

Quad 1 Quad 5 Quad 2 Quad 6 Quad 3 Quad 7 Quad 4 Quad 8

915101811211224

Notes:

Memory quads 1-12 must be populated.
Memory quads on the same processor must be populated with CDIMMs of the same capacity.

Chapter 2. Architecture and technical overview 53

5137ch02.fm Draft Document for Review October 14, 2014 10:19 am

Table 2-8 shows the CDIMM plugging order for a Power E880 with a four system nodes:

Table 2-8 Optimal CDIMM memory quad placement for four system nodes

System Node 1

Processor 0 Processor 2 Processor 3 Processor 1

Quad 1 Quad 5 Quad 2 Quad 6 Quad 3 Quad 7 Quad 4 Quad 8

1172213254 29

System Node 2

Processor 0 Processor 2 Processor 3 Processor 1

Quad 1 Quad 5 Quad 2 Quad 6 Quad 3 Quad 7 Quad 4 Quad 8

5186227268 30

System Node 3

Processor 0 Processor 2 Processor 3 Processor 1

Quad 1 Quad 5 Quad 2 Quad 6 Quad 3 Quad 7 Quad 4 Quad 8

919102311271231

Processor 0 Processor 2 Processor 3 Processor 1

Quad 1 Quad 5 Quad 2 Quad 6 Quad 3 Quad 7 Quad 4 Quad 8

13 20 14 24 15 28 16 32

Notes:

Memory quads 1-16 must be populated.
Memory quads on the same processor must be populated with CDIMMs of the same capacity.

2.3.3 Memory activation

All the memory CDIMMs are capable of capacity upgrade on demand and must have a
minimum of 50% of its physical capacity activated. For example, the minimum installed
memory for both Power E870 and Power E880 is 256 GB of RAM, whereas they can have a
minimum of 128 GB of RAM active.

There are two activation types that can be used to accomplish this:

򐂰 Static memory activations: Memory activations that are exclusive for a single server
򐂰 Mobile memory activations: Memory activations that can be moved from server to server

in a power enterprise pool.

Both types of memory activations can co-reside in the same system, as long as at least 25%
of the memory activations are static. This leads to a maximum of 75% of the memory
activations as mobile.

System Node 4

54 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch02.fm

1024 GB

512 GB

768 GB

256 GB

Installed memor y

Minimum active

Minimum Static active

Maximum Mobile active

50% of total memory

75% of installed memory

25% of installed memory

Without Power Enterprise Pools

With Power Enterprise Pools

On Figure 2-15 on page 55 there is an example of the minimum required activations for a
system with 1 TB of installed memory.

Figure 2-15 Example of the minimum required activations for a system with 1 TB of installed memory

2.3.4 Memory throughput

The granularity for static memory activation is 1 GB, while for mobile memory activation the
granularity is 100 GB. On Table 2-9 there is a list of the feature codes that can be used to
achieve the desired number of activations:

Table 2-9 Static and mobile memory activation feature codes

Feature code Feature description Amount of memory Type of activation

EMA5 1 GB Memory activation 1 GB Static

EMA6 100 GB Memory activation 100 GB Static

EMA9 100 GB Mobile memory activation 100 GB Mobile

Static memory activations can be converted to mobile memory activations after system
installation. In order to enable mobile memory activations, the systems must be part of a
power enterprise pool and have feature code #EB35 configured. For more information on
power enterprise pools, please see 2.4.2, “Power enterprise pools and mobile capacity on
demand (Mobile CUoD)” on page 62.

The peak memory and I/O bandwidths per system node have increased over 300% compared
to previous POWER7 servers, providing the next generation of data intensive applications
with a platform capable of handling the needed amount of data.

Power E870

Table 2-10 shows the maximum bandwidth estimates for a single core on the Power E870
system.

Chapter 2. Architecture and technical overview 55

5137ch02.fm Draft Document for Review October 14, 2014 10:19 am

Table 2-10 Power E870 single core bandwidth estimates

Single core Power E870 Power E870

1 core @ 4.024 GHz 1 core @ 4.190 GHz

L1 (data) cache 193.15 GBps 201.12 GBps

L2 cache 193.15 GBps 201.12 GBps

L3 cache 257.54 GBps 268.16 GBps

The bandwidth figures for the caches are calculated as follows:
򐂰 L1 cache: In one clock cycle, two 16-byte load operations and one 16-byte store operation

can be accomplished. The value varies depending on the clock of the core and the formula
is as follows:

– 4.024 GHz Core: (2 * 16 B + 1 * 16 B) * 4.024 GHz = 193.15 GBps
– 4.190 GHz Core: (2 * 16 B + 1 * 16 B) * 4.190 GHz = 201.12 GBps

򐂰 L2 cache: In one clock cycle, one 32-byte load operation and one 16-byte store operation

can be accomplished. The value varies depending on the clock of the core and the formula
is as follows:

– 4.024 GHz Core: (1 * 32 B + 1 * 16 B) * 4.024 GHz = 193.15 GBps
– 4.190 GHz Core: (1 * 32 B + 1 * 16 B) * 4.190 GHz = 201.12 GBps

򐂰 L3 cache: In one clock cycle, one 32-byte load operation and one 32-byte store operation

can be accomplished. The value varies depending on the clock of the core and the formula
is as follows:

– 4.024 GHz Core: (1 * 32 B + 1 * 32 B) * 4.024 GHz = 257.54 GBps
– 4.190 GHz Core: (1 * 32 B + 1 * 32 B) * 4.190 GHz = 286.16 GBps

For each system node of a Power E870 populated with four processors and all its memory
CDIMMs filled, the overall bandwidths are shown in Table 2-11.

Table 2-11 Power E870 system node bandwidth estimates

System node bandwidths Power E870 Power E870

32 cores @ 4.024 GHz 40 cores @ 4.190 GHz

L1 (data) cache 6,181 GBps 8,045 GBps

L2 cache 6,181 GBps 8,045 GBps

L3 cache 8,241 GBps 10,726 GBps

Total Memory 922 GBps 922 GBps

PCIe Interconnect 252.064 GBps 252.064 GBps

Intra-node buses
(two system nodes)

922 GBps 922 GBps

PCIe Interconnect: Each POWER8 processor has 32 PCIe lanes running at 7.877Gbps
full-duplex. The bandwidth formula is calculated as follows:

32 lanes * 4 processors * 7.877 Gbps * 2 = 252.064 GBps

Rounding: The bandwidths listed here may appear slightly differently in other materials
due to rounding of some figures.

56 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch02.fm

For the entire Power E870 system populated with two system nodes, the overall bandwidths
are what is shown in Table 2-12 on page 57.

Table 2-12 Power E870 total bandwidth estimates

Total bandwidths Power E870 Power E870

64 cores @ 4.024 GHz 80 cores @ 4.190 GHz

L1 (data) cache 12,362 GBps 16,090 GBps

L2 cache 12,362 GBps 16,090 GBps

L3 cache 16,484 GBps 21,453 GBps

Total Memory 1,844 GBps 1,844 GBps

PCIe Interconnect 504.128 GBps 504.128 GBps

Inter-node buses (two
system nodes)

Intra-node buses (two
system nodes)

307 GBps 307 GBps

1,844 GBps 1,844 GBps

Power E880

Table 2-13 shows the maximum bandwidth estimates for a single core on the Power E880
system.

Table 2-13 Power E880 single core bandwidth estimates

Single core Power E880

1 core @ 4.350 GHz

L1 (data) cache 208.80 GBps

L2 cache 208.80 GBps

L3 cache 278.40 GBps

The bandwidth figures for the caches are calculated as follows:
򐂰 L1 cache: In one clock cycle, two 16-byte load operations and one 16-byte store operation

can be accomplished. The value varies depending on the clock of the core and the formula
is as follows:

– 4.350 GHz Core: (2 * 16 B + 1 * 16 B) * 4.350 GHz = 208.80 GBps

򐂰 L2 cache: In one clock cycle, one 32-byte load operation and one 16-byte store operation

can be accomplished. The value varies depending on the clock of the core and the formula
is as follows:

– 4.350 GHz Core: (1 * 32 B + 1 * 16 B) * 4.350 GHz = 208.80 GBps

򐂰 L3 cache: In one clock cycle, one 32-byte load operation and one 32-byte store operation

can be accomplished. The value varies depending on the clock of the core and the formula
is as follows:

– 4.350 GHz Core: (1 * 32 B + 1 * 32 B) * 4.350 GHz = 278.40 GBps

For each system node of a Power E880 populated with four processors and all its memory
CDIMMs filled, the overall bandwidths are shown in Table 2-14 on page 58.

Chapter 2. Architecture and technical overview 57

5137ch02.fm Draft Document for Review October 14, 2014 10:19 am

Table 2-14 Power E880 system node bandwidth estimates

System node bandwidths Power E880

32 cores @ 4.350 GHz

L1 (data) cache 6,682 GBps

L2 cache 6,682 GBps

L3 cache 8,909 GBps

Total Memory 922 GBps

PCIe Interconnect 252.064 GBps

Intra-node buses
(two system nodes)

922 GBps

PCIe Interconnect: Each POWER8 processor has 32 PCIe lanes running at 7.877 Gbps
full-duplex. The bandwidth formula is calculated as follows:

32 lanes * 4 processors * 7.877Gbps * 2 = 252.064 GBps

Rounding: The bandwidths listed here may appear slightly differently in other materials
due to rounding of some figures.

For the entire Power E880 system populated with four system nodes, the overall bandwidths
are shown in Table 2-15.

Table 2-15 Power E880 total bandwidth estimates

Total bandwidths Power E880

128 cores @ 4.350 GHz

L1 (data) cache 26,726 GBps

L2 cache 26,726 GBps

L3 cache 35,635 GBps

Total Memory 3,688 GBps

PCIe Interconnect 1008.256 GBps

Inter-node buses (four
system nodes)

Intra-node buses (four
system nodes)

Note: At the time of writing, the Power E880 supports only two system nodes per server.

2.3.5 Active Memory Mirroring

The Power E870 and Power E880 systems have the ability to provide mirroring of the
hypervisor code across multiple memory CDIMMs. If a CDIMM that contains the hypervisor
code develops an uncorrectable error, its mirrored partner will enable the system to continue
to operate uninterrupted.

58 IBM Power Systems E870 and E880 Technical Overview and Introduction

307 GBps

3,688 GBps

Draft Document for Review October 14, 2014 10:19 am 5137ch02.fm

Active Memory Mirroring (AMM) is included with all Power E870 and Power E880 systems at
no additional charge. It can be enabled, disabled, or re-enabled depending on the user’s
requirements.

The hypervisor code logical memory blocks will be mirrored on distinct CDIMMs to allow for
more usable memory. There is no specific CDIMM that hosts the hypervisor memory blocks
so the mirroring is done at the logical memory block level, not at the CDIMM level. To enable
the AMM feature it is mandatory that the server has enough free memory to accommodate
the mirrored memory blocks.

Besides the hypervisor code itself, other components that are vital to the server operation are
also mirrored:

򐂰 Hardware page tables (HPTs), responsible for tracking the state of the memory pages

assigned to partitions

򐂰 Translation control entities (TCEs), responsible for providing I/O buffers for the

partition’s communications

򐂰 Memory used by the hypervisor to maintain partition configuration, I/O states, virtual I/O

information, and partition state

It is possible to check whether the Active Memory Mirroring option is enabled and change its
current status through HMC, under the Advanced Tab on the CEC Properties panel
(Figure 2-16).

Figure 2-16 CEC Properties panel on an HMC

After a failure on one of the CDIMMs containing hypervisor data occurs, all the server
operations remain active and flexible service processor (FSP) will isolate the failing CDIMMs.
Systems stay in the partially mirrored state until the failing CDIMM is replaced.

There are components that are not mirrored because they are not vital to the regular server
operations and require a larger amount of memory to accommodate its data:

򐂰 Advanced Memory Sharing Pool
򐂰 Memory used to hold the contents of platform dumps

Chapter 2. Architecture and technical overview 59

5137ch02.fm Draft Document for Review October 14, 2014 10:19 am

Partition data: Active Memory Mirroring will not mirror partition data. It was designed to

mirror only the hypervisor code and its components, allowing this data to be protected
against a DIMM failure

With AMM, uncorrectable errors in data that are owned by a partition or application are
handled by the existing Special Uncorrectable Error handling methods in the hardware,
firmware, and operating system.

2.3.6 Memory Error Correction and Recovery

The memory has error detection and correction circuitry is designed such that the failure of
any one specific memory module within an ECC word can be corrected without any other
fault.

In addition, a spare DRAM per rank on each memory port provides for dynamic DRAM device
replacement during runtime operation. Also, dynamic lane sparing on the DMI link allows for
repair of a faulty data lane.

Other memory protection features include retry capabilities for certain faults detected at both
the memory controller and the memory buffer.

Memory is also periodically scrubbed to allow for soft errors to be corrected and for solid
single-cell errors reported to the hypervisor, which supports operating system deallocation of
a page associated with a hard single-cell fault.

For more details on Memory RAS, see 4.3.10, “Memory protection” on page 153.

2.3.7 Special Uncorrectable Error handling

Special Uncorrectable Error (SUE) handling prevents an uncorrectable error in memory or
cache from immediately causing the system to terminate. Rather, the system tags the data
and determines whether it will ever be used again. If the error is irrelevant, it does not force a
checkstop. If the data is used, termination can be limited to the program/kernel or hypervisor
owning the data, or freeze of the I/O adapters controlled by an I/O hub controller if data is to
be transferred to an I/O device.

2.4 Capacity on Demand

Several types of Capacity on Demand (CoD) offerings are optionally available on the
Power 870 and Power E880 servers to help meet changing resource requirements in an
on-demand environment, by using resources that are installed on the system but that are not
activated.

2.4.1 Capacity Upgrade on Demand (CUoD)

Power E870 and Power E880 systems include a number of active processor cores and
memory units. They can also include inactive processor cores and memory units. Active
processor cores or memory units are processor cores or memory units that are already
available for use on your server when it comes from the manufacturer. Inactive processor
cores or memory units are processor cores or memory units that are included with your
server, but not available for use until you activate them. Inactive processor cores and memory

60 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch02.fm

0

10

20

30

40

50

60

70

80

Without CuOD

With CuOD

Time

Cores

Projected

Core Activations

units can be permanently activated by purchasing an activation feature called Capacity
Upgrade on Demand (CUoD) and entering the provided activation code on your server.

With the CUoD offering, you can purchase additional static processor or memory capacity
and dynamically activate them when needed, without requiring you to restart your server or
interrupt your business. All the static processor or memory activations are restricted to a
single server.

Capacity Upgrade on Demand can have several applications to allow for a more flexible
environment. One of its benefits is to allow for a given company to reduce the initial
investment on a system. Traditional projects using other technologies require that the a
system is acquired with all the resources available to support the whole lifecycle of the
project. This might incur in costs that would only be necessary on later stages of the project,
usually with impacts on software licensing costs and software maintenance.

By using Capacity Upgrade on Demand a company could start with a system with enough
installed resources to support the whole project lifecycle but only with enough active
resources necessary for the initial project phases. More resources could be added along with
the project, adjusting the hardware platform with the project needs. This would allow for a
company to reduce the initial investment in hardware and only acquire software licenses that
are needed on each project phase, reducing the Total Cost of Ownership and Total Cost of
Acquisition of the solution. Figure 2-17 shows a comparison between two scenarios: a fully
activated system versus a system with CUoD resources being activated along with the project
timeline.

Figure 2-17 Active cores scenarios comparison during a project lifecycle

The Table 2-16 on page 62 lists the static processor activation features that are available for
the Power E870 and Power E880.

Chapter 2. Architecture and technical overview 61

5137ch02.fm Draft Document for Review October 14, 2014 10:19 am

Table 2-16 Power Systems Capacity Upgrade on Demand static processor activation features

System Processor feature CUoD processor core activation

feature

Power E870 #EPBA (4.02 GHz Processor Card) #EPBJ

Power E870 #EPBC (4.19 GHz Processor Card) #EPBL

Power E880 #EPBB (4.35 GHz Processor Card) #EPBK

The Table 2-17 lists the static memory activation features that are available for the
Power E870 and Power E880.

Table 2-17 Power Systems Capacity Upgrade on Demand static memory activation features

System Description Feature code

Power E870, Power E880 Activation of 1 GB DDR3 POWER8 memory #EMA5

Power E870, Power E880 Activation of 100 GB DDR3 POWER8 memory #EMA6

2.4.2 Power enterprise pools and mobile capacity on demand (Mobile CUoD)

While static activations are valid for a single system, some customers could benefit from
moving processor and memory activations among different servers due to workload
rebalance or disaster recovery.

IBM power enterprise pools, is a technology for dynamically sharing processor and memory
activations among a group (or pool) of IBM Power Systems servers. Using Mobile Capacity on
Demand (CoD) activation codes, the systems administrator can perform tasks without
contacting IBM.

Two types of power enterprise pools are available:

򐂰 Power 770 (9117-MMD) and Power E870 (9119-MME) class systems
򐂰 Power 780 (9117-MHD), Power 795 (9119-FHB), and Power E870 (9119-MHE) class

systems

Each pool type can support systems with different clock speeds or processor generations.

The basic rules for the Mobile Capacity on Demand follows:
򐂰 The Power 770 and Power 780 systems require a minimum of four static processor

activations.

򐂰 The Power 870 and Power 880 require a minimum of eight static processor activations.
򐂰 The Power 795 requires a minimum of 24 static processor activations or 25% of the

installed processor capacity whichever is bigger.

򐂰 For all systems, 25% of the installed processor capacity must have static activations.

All the systems in a pool must be managed by the same HMC or by the same pair of
redundant HMCs. If redundant HMCs are used, the HMCs must be connected to a network so
that they can communicate with each other. The HMCs must have at least 2 GB of memory.

An HMC can manage multiple power enterprise pools and can also manage systems that are
not part of a power enterprise pool. Systems can belong to only one power enterprise pool at
a time. Powering down an HMC does not limit the assigned resources of participating
systems in a pool but does limit the ability to perform pool change operations.

62 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch02.fm

After a power enterprise pool is created, the HMC can be used to perform the following
functions:

򐂰 Mobile CoD processor and memory resources can be assigned to systems with inactive

resources. Mobile CoD resources remain on the system to which they are assigned until
they are removed from the system.

򐂰 New systems can be added to the pool and existing systems can be removed from the

pool.

򐂰 New resources can be added to the pool or existing resources can be removed from the

pool.

򐂰 Pool information can be viewed, including pool resource assignments, compliance, and

history logs.

In order for the Mobile activation features to be configured it is necessary that a a power
enterprise pool is registered with IBM as well as the systems which are going to be included
as members of the pool. Also it is necessary that the systems have the feature code #EB35
for mobile enablement configured and the required contracts must be in place.

The Table 2-18 lists the mobile processor activation features that are available for the
Power E870 and Power E880.

Table 2-18 Mobile processor activation features

System Description CUoD mobile processor core

Power E870 1-Core Mobile activation #EP2S

Power E880 1-Core Mobile activation #EP2T

The Table 2-19 lists the mobile memory activation features that are available for the
Power E870 and Power E880.

Table 2-19 Mobile memory activation features

System Description Feature code

Power E870, Power E880 100 GB Mobile memory activation #EMA4

For more information on power enterprise pools, please see Redpaper “power enterprise
pools on IBM Power Systems”.

http://www.redbooks.ibm.com/Redbooks.nsf/RedbookAbstracts/redp5101.html?Open

2.4.3 Elastic Capacity on Demand (Elastic CoD)

With the elastic CoD offering, you can temporarily activate and deactivate processor cores
and memory units to help meet the demands of business peaks such as seasonal activity,
period-end, or special promotions. Elastic CoD was previously called On/Off CoD. When you
order an elastic CoD feature, you receive an enablement code that allows a system operator
to make requests for additional processor and memory capacity in increments of one
processor day or 1 GB memory day. The system monitors the amount and duration of the
activations. Both prepaid and post-pay options are available.

activation feature

Charges are based on usage reporting that is collected monthly. Processors and memory
may be activated and turned off an unlimited number of times, when additional processing
resources are needed.

Chapter 2. Architecture and technical overview 63

5137ch02.fm Draft Document for Review October 14, 2014 10:19 am

This offering provides a system administrator an interface at the HMC to manage the
activation and deactivation of resources. A monitor that resides on the server records the
usage activity. This usage data must be sent to IBM on a monthly basis. A bill is then
generated based on the total amount of processor and memory resources utilized, in
increments of Processor and Memory (1 GB) Days.

New to both Power E870 and Power E880 are 90-day temporary elastic CoD processor and
memory enablement features. These features enable a system to temporarily activate all
inactive processor and memory CoD resources for a maximum of 90 days before ordering
another temporary elastic enablement feature code is required.

Before using temporary capacity on your server, you must enable your server. To enable, an
enablement feature (MES only) must be ordered and the required contracts must be in place.

If a Power E870 or Power E880 server uses the IBM i operating system in addition to any
other supported operating system on the same server, the client must inform IBM which
operating system caused the temporary elastic CoD processor usage so that the correct
feature can be used for billing.

The features that are used to order enablement codes and support billing charges on the
Power E870 and Power E880 are described in 1.4, “System features” on page 9 and 1.4.7,
“Memory features” on page 16.

The elastic CoD process consists of three steps: enablement, activation, and billing.
򐂰 Enablement

Before requesting temporary capacity on a server, you must enable it for elastic CoD. To
do this, order an enablement feature and sign the required contracts. IBM will generate an
enablement code, mail it to you, and post it on the web for you to retrieve and enter on the
target server.

A

processor enablement code allows you to request up to 360 processor days of

temporary capacity. If the 360 processor-day limit is reached, place an order for another
processor enablement code to reset the number of days that you can request back to 360.

A

memory enablement code lets you request up to 999 memory days of temporary

capacity. If you reach the limit of 999 memory days, place an order for another memory
enablement code to reset the number of allowable days you can request back to 999.

򐂰 Activation requests

When elastic CoD temporary capacity is needed, use the HMC menu for On/Off CoD.
Specify how many inactive processors or gigabytes of memory are required to be
temporarily activated for some number of days. You are billed for the days requested,
whether the capacity is assigned to partitions or remain in the shared processor pool.

At the end of the temporary period (days that were requested), you must ensure that the
temporarily activated capacity is available to be reclaimed by the server (not assigned to
partitions), or you are billed for any unreturned processor days.

򐂰 Billing

The contract, signed by the client before receiving the enablement code, requires the
elastic CoD user to report billing data at least once a month (whether or not activity
occurs). This data is used to determine the proper amount to bill at the end of each billing
period (calendar quarter). Failure to report billing data for use of temporary processor or
memory capacity during a billing quarter can result in default billing equivalent to 90
processor days of temporary capacity.

64 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch02.fm

For more information about registration, enablement, and usage of elastic CoD, visit the
following location:

http://www.ibm.com/systems/power/hardware/cod

2.4.4 Utility Capacity on Demand (Utility CoD)

Utility CoD automatically provides additional processor performance on a temporary basis
within the shared processor pool.

With Utility CoD, you can place a quantity of inactive processors into the server’s shared
processor pool, which then becomes available to the pool's resource manager. When the
server recognizes that the combined processor utilization within the shared processor pool
exceeds 100% of the level of base (purchased and active) processors that are assigned
across uncapped partitions, then a Utility CoD processor minute is charged and this level of
performance is available for the next minute of use.

If additional workload requires a higher level of performance, the system automatically allows
the additional Utility CoD processors to be used, and the system automatically and
continuously monitors and charges for the performance needed above the base (permanent)
level.

Registration and usage reporting for utility CoD is made using a public website and payment
is based on reported usage. Utility CoD requires PowerVM Standard Edition or PowerVM
Enterprise Edition to be active.

If a Power E870 or Power E880 server uses the IBM i operating system in addition to any
other supported operating system on the same server, the client must inform IBM which
operating system caused the temporary Utility CoD processor usage so that the correct
feature can be used for billing.

For more information regarding registration, enablement, and use of Utility CoD, visit the
following location:

http://www.ibm.com/systems/support/planning/capacity/index.html

2.4.5 Trial Capacity on Demand (Trial CoD)

A standard request for Trial CoD requires you to complete a form including contact information
and vital product data (VPD) from your Power E870 or Power E880 system with inactive CoD
resources.

A standard request activates two processors or 4 GB of memory (or both two processors and
4 GB of memory) for 30 days. Subsequent standard requests can be made after each
purchase of a permanent processor activation. An HMC is required to manage Trial CoD
activations.

An

exception request for Trial CoD requires you to complete a form including contact

information and VPD from your Power E870 or Power E880 system with inactive CoD
resources. An exception request will activate all inactive processors or all inactive memory (or
all inactive processor and memory) for 30 days. An exception request can be made only one
time over the life of the machine. An HMC is required to manage Trial CoD activations.

To request either a Standard or an Exception Trial, visit the following location:

https://www-912.ibm.com/tcod_reg.nsf/TrialCod?OpenForm

Chapter 2. Architecture and technical overview 65

5137ch02.fm Draft Document for Review October 14, 2014 10:19 am

I/O Backplane

PCIe Gen3 x16 slot

PCIe Gen3 x16 slot

PCIe Gen3 x16 slot

PCIe Gen3 x16 slot

PCIe Gen3 x16 slot

PCIe Gen3 x16 slot

PCIe Gen3 x16 slot

PCIe Gen3 x16 slot

P8 (0)

PHB 1

PHB 0

SMP

X Bus

SMP

A Bus

Memory

Controller 0

Memory

Controller 1

P8 (2)

PHB 1

PHB 0

SMP

X Bus

SMP

A Bus

Memory

Controller 0

Memory

Controller 1

P8 (3)

PHB 1

PHB 0

SMP

X Bus

SMP

A Bus

Memory

Controller 0

Memory

Controller 1

P8 (1)

Memory

Controller 0

Memory

Controller 1

PHB 1

PHB 0

SMP

X Bus

SMP

A Bus

2.4.6 Software licensing and CoD

For software licensing considerations with the various CoD offerings, see the most recent
revision of the Power Systems Capacity on Demand User’s Guide:

http://www.ibm.com/systems/power/hardware/cod

2.5 System bus

This section provides additional information related to the internal buses.

2.5.1 PCI Express Gen3

The internal I/O subsystem on the Power E870 and Power E880 is connected to the PCIe
Controllers on a POWER8 processor in the system. Each POWER8 processor module has
two buses that have 16 PCIe lanes each (for a total of 32 PCIe lanes) running at 7.877 Gbps
full-duplex and provides 31.508 GBps of I/O connectivity to the PCIe slots. A diagram with the
connections can be seen in Figure 2-18.

Figure 2-18 System nodes PCIe slots directly attached to PCIe controllers on POWER8 chips

Besides the slots directly attached to the processors PCI Gen3 controllers, the systems also

66 IBM Power Systems E870 and E880 Technical Overview and Introduction

allow for additional PCIe adapters on external PCIe Expansion Drawers and disks on external
drawers connected through PCIe SAS adapters.

Figure 2-19 on page 67 shows a diagram with the I/O connectivity options available for the
Power E870 and Power E880. The system nodes allow for eight PCIe Gen3 x16 slots.
Additional slots can be added by attaching PCIe Expansion Drawers and SAS disks can be

Draft Document for Review October 14, 2014 10:19 am 5137ch02.fm

PCIe Gen3 x16

PCIe Adapter for

Expansion Dr awer

PCIe Gen3 x16

PCIe Gen3 x16

PCIe Gen3 x16

PCIe Gen3 x16

PCIe Gen3 x16

SAS Adapter

System Node

EXP24S SFF Gen2 Drawer

PCIe Expansion Drawer

P

C

I

e

G

e

n

3

x

8

P

C

I

e

G

e

n

3

x

8

P

C

I

e

G

e

n

3

x

8

P

C

I

e

G

e

n

3

x

8

P

C

I

e

G

e

n

3

x

8

P

C

I

e

G

e

n

3

x

8

P

C

I

e

G

e

n

3

x

8

P

C

I

e

G

e

n

3

x

8

P

C

I

e

G

e

n

3

x

8

P

C

I

e

G

e

n

3

x

8

P

C

I

e

G

e

n

3

x

8

S

A

S

A

d

a

p

t

e

r

Disks

PCIe Adapters

EXP24S SFF Gen2 Drawer

CPU

1

CPU

0

CPU

2

CPU

3

attached to EXP24S SFF Gen2 Drawers. The EXP24S can be either attached to SAS
adapters on the system nodes or on the PCIe Expansion Drawer.

For a list of adapters and their supported slots, please see 2.7, “PCI adapters” on page 70.

I

2.5.2 Service Processor Bus

Figure 2-19 I/O connectivity options available for Power E870 and Power E880

Disk support: There is no support for disks directly installed on the system nodes and
PCIe Expansion Drawers. If directly attached SAS disk are required, they need to be
installed in a SAS disk drawer and connected to a supported SAS controller in one of the
PCIe slots.

For more information on PCIe Expansion Drawers, please see 2.9.1, “PCIe Gen3 I/O
expansion drawer” on page 77.

The redundant service processor bus connectors are located on the rear of the control unit
and the system nodes. All the service processor (SP) communication between the control unit
and the system nodes flows though these cables.

Unlike the previous generations where a given pair of enclosures would host the service
processors, on Power E870 and Power E880 as a standard, redundant service processor
cards are installed on the control unit as well as redundant clock cards.

The cables used to provide communications between the control units and system nodes
depend on the amount of system nodes installed. When a system node is added, a new set of
cables is also added.

Chapter 2. Architecture and technical overview 67

5137ch02.fm Draft Document for Review October 14, 2014 10:19 am

The cables necessary for each system node have been grouped under a single feature code,
allowing for an easier configuration. Each cable set includes a pair of FSP cables, a pair of
clock cables, and when applicable SMP cables and UPIC cables. Table 2-20 shows a list of
the feature codes available:

Table 2-20 Features for cable sets

Feature code Description

ECCA System node to system control unit cable set for drawer 1

ECCB System node to system control unit cable set for drawer 2

ECCC System node to system control unit cable set for drawer 3

ECCD System node to system control unit cable set for drawer 4

Cable sets feature codes are incremental and depend on the number of installed drawers as
follows:

򐂰 1 system node: #ECCA
򐂰 2 system nodes: #ECCA and #ECCB
򐂰 3 system nodes: #ECCA, #ECCB, and #ECCC
򐂰 4 system nodes: #ECCA, #ECCB, #ECCC, and #ECCD

Note: As the time of writing, up to two system nodes are supported for Power E870 and
Power E880.

System connection topology is shown in 2.1, “Logical diagrams” on page 32.

2.6 Internal I/O Subsystem

The internal I/O subsystem resides on the I/O planar, which supports eight PCIe Gen3 x16
slots. All PCIe slots are hot-pluggable and enabled with enhanced error handling (EEH). In
the unlikely event of a problem, EEH-enabled adapters respond to a special data packet that
is generated from the affected PCIe slot hardware by calling system firmware, which
examines the affected bus, allows the device driver to reset it, and continues without a system
reboot. For more information about RAS on the I/O buses, see 4.3.11, “I/O subsystem
availability and Enhanced Error Handling” on page 154.

Table 2-21 lists the slot configuration of a Power E870 and Power E880 system nodes.

Table 2-21 Slot configuration and capabilities

Slot Location code Slot Type CAPI capable

Slot 1 P1-C1 PCIe Gen3 x16 No Yes

Slot 2 P1-C2 PCIe Gen3 x16 Yes Yes

a

SRIOV capable

Slot 3 P1-C3 PCIe Gen3 x16 No Yes

Slot 4 P1-C4 PCIe Gen3 x16 Yes Yes

Slot 5 P1-C5 PCIe Gen3 x16 No Yes

Slot 6 P1-C6 PCIe Gen3 x16 Yes Yes

Slot 7 P1-C7 PCIe Gen3 x16 No Yes

68 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch02.fm

I/O Backplane

PCIe Gen3 x16 slot

PCIe Gen3 x16 slot

PCIe Gen3 x16 slot

PCIe Gen3 x16 slot

PCIe Gen3 x16 slot

PCIe Gen3 x16 slot

PCIe Gen3 x16 slot

PCIe Gen3 x16 slot

P8 (0)

PHB 1

PHB 0

SMP

X Bus

SMP

A Bus

Memory

Controller 0

Memory

Controller 1

P8 (2)

PHB 1

PHB 0

SMP

X Bus

SMP

A Bus

Memory

Controller 0

Memory

Controller 1

P8 (3)

PHB 1

PHB 0

SMP

X Bus

SMP

A Bus

Memory

Controller 0

Memory

Controller 1

P8 (1)

Memory

Controller 0

Memory

Controller 1

PHB 1

PHB 0

SMP

X Bus

SMP

A Bus

P1-C1

P1-C2

P1-C3

P1-C4

P1-C5

P1-C6

P1-C7

P1-C8

Front

Rear

System Node top view

Slot Location code Slot Type CAPI capable

a

SRIOV capable

Slot 8 P1-C8 PCIe Gen3 x16 Yes Yes

a. At the time of writing there are no supported CAPI adapters for the Power E870 and E880

system unit.

The physical location of the slots can be seen on Figure 2-20.

2.6.1 Blind-swap cassettes

Figure 2-20 System node top view and PCIe slot location codes

The Power E870 and Power E880 use a next generation blind-swap cassette to manage the
installation and removal of PCIe adapters. This mechanism requires an interposer card that
allows the PCIe adapters to plug in vertically to the system, allows more airflow through the
cassette, and allows for faster hot swap procedures. Cassettes can be installed and removed
without removing the system nodes or PCIe expansion drawers from the rack.

Chapter 2. Architecture and technical overview 69

5137ch02.fm Draft Document for Review October 14, 2014 10:19 am

FSP 1 FSP 2

HMC1

HMC2

HMC1

HMC2

USB

2.6.2 System ports

The system nodes do not have integrated ports. All networking and storage for the virtual
machines must pre provided via PCIe adapters installed in standard PCIe slots.

The system control unit has one USB Port dedicated to the DVD drive and 4 Ethernet ports
used for HMC communications. There is no serial port so an HMC is mandatory for system
management. The FSP’s virtual console will be on the HMC.

The location of the USB and HMC Ethernet ports can be seen on Figure 2-21

Figure 2-21 Physical location of the USB and HMC ports on the system control unit

The connection and usage of the DVD can be seen on detail on 2.8.1, “DVD” on page 76.

2.7 PCI adapters

This section covers the types and functions of the PCI cards supported by IBM Power E870
and IBM Power E880 systems.

2.7.1 PCI Express (PCIe)

PCIe uses a serial interface and allows for point-to-point interconnections between devices
(using a directly wired interface between these connection points). A single PCIe serial link is
a dual-simplex connection that uses two pairs of wires, one pair for transmit and one pair for
receive, and can transmit only one bit per cycle. These two pairs of wires are called a
PCIe link can consist of multiple lanes. In such configurations, the connection is labelled as
x1, x2, x8, x12, x16, or x32, where the number is effectively the number of lanes.

The PCIe interfaces supported on this server are PCIe Gen3, capable of 16 GBps simplex
(32 GBps duplex) on a single x16 interface. PCIe Gen3 slots also support previous
generations (Gen2 and Gen1) adapters, which operate at lower speeds, according to the
following rules:

򐂰 Place x1, x4, x8, and x16 speed adapters in same connector size slots first, before mixing

adapter speed with connector slot size.

lane. A

򐂰 Adapters with smaller speeds are allowed in larger sized PCIe connectors but larger

speed adapters are not compatible in smaller connector sizes (i.e. a x16 adapter cannot
go in an x8 PCIe slot connector).

70 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch02.fm

IBM POWER8 processor-based servers can support two different form factors of PCIe
adapters:

򐂰 PCIe low profile (LP) cards, which are used with system node PCIe slots.
򐂰 PCIe full height and full high cards are used in the PCIe Gen3 I/O expansion drawer

(#EMX0)

Low-profile PCIe adapter cards are supported only in low-profile PCIe slots, and full-height
and full-high cards are supported only in full-high slots.

Before adding or rearranging adapters, use the System Planning Tool to validate the new
adapter configuration. For more information, see the System Planning Tool website:

http://www.ibm.com/systems/support/tools/systemplanningtool/

If you are installing a new feature, ensure that you have the software that is required to
support the new feature and determine whether there are any existing update prerequisites to
install. To do this, use the IBM prerequisite website:

https://www-912.ibm.com/e_dir/eServerPreReq.nsf

The following sections describe the supported adapters and provide tables of orderable
feature numbers. The tables indicate operating system support (AIX, IBM i, and Linux) for
each of the adapters.

2.7.2 LAN Adapters

To connect the Power E870 and Power E880 servers to a local area network (LAN), you can
use the LAN adapters that are supported in the PCIe slots of the system. Table 2-22 lists the
available LAN adapters.

Table 2-22 Available LAN adapters

Feature
Code

5260 576F PCIe2 LP 4-port 1 GbE Adapter CEC AIX, IBM i,

5287 5287 PCIe2 2-port 10 GbE SR Adapter I/O drawer AIX, Linux

5717 5717 4-Port 10/100/1000 Base-TX PCI Express

5767 5767 2-Port 10/100/1000 Base-TX Ethernet PCI

5768 5768 2-Port Gigabit Ethernet-SX PCI Express

5769 5769 10 Gigabit Ethernet-SR PCI Express Adapter I/O drawer AIX, IBM i,

5772 576E 10 Gigabit Ethernet-LR PCI Express Adapter I/O drawer AIX, IBM i,

CCIN Description Placement OS support

Linux

I/O drawer AIX, Linux

Adapter

I/O drawer AIX, IBM i,

Express Adapter

I/O drawer AIX, IBM i,

Adapter

Linux

Linux

Linux

Linux

5899 576F PCIe2 4-port 1 GbE Adapter I/O drawer AIX, IBM i,

EC28 EC27 PCIe2 2-Port 10 GbE RoCE SFP+ Adapter I/O drawer AIX, Linux

EC29 EC29 PCIe2 LP 2-Port 10 GbE RoCE SR Adapter CEC AIX, Linux

Linux

Chapter 2. Architecture and technical overview 71

5137ch02.fm Draft Document for Review October 14, 2014 10:19 am

Feature
Code

EC2J EC2G PCIe2 2-port 10 GbE SFN6122F Adapter I/O drawer Linux

EC30 EC29 PCIe2 2-Port 10 GbE RoCE SR Adapter I/O drawer AIX, Linux

EC3A 57BD PCIe3 LP 2-Port 40 GbE NIC RoCE QSFP+

EC3B 57B6 PCIe3 2-Port 40 GbE NIC RoCE QSFP+

EN0S 2CC3 PCIe2 4-Port (10Gb+1 GbE) SR+RJ45

EN0U 2CC3 PCIe2 4-port (10Gb+1 GbE) Copper

EN0W 2CC4 PCIe2 2-port 10/1 GbE BaseT RJ45 Adapter I/O drawer AIX, Linux

EN0X 2CC4 PCIe2 LP 2-port 10/1 GbE BaseT RJ45

CCIN Description Placement OS support

Adapter

Adapter

Adapter

SFP+RJ45 Adapter

Adapter

2.7.3 Graphics accelerator adapters

Table 2-23 lists the available graphics accelerator adapters. An adapter can be configured to
operate in either 8-bit or 24-bit color modes. The adapter supports both analog and digital
monitors.

CEC AIX, Linux

I/O drawer AIX, Linux

I/O drawer AIX, Linux

I/O drawer AIX, Linux

CEC AIX, Linux

Table 2-23 Available graphics accelerator adapters

Feature
Code

5748 5748 POWER GXT145 PCI Express Graphics

EC41 PCIe2 LP 3D Graphics Adapter CEC Linux

2.7.4 SAS adapters

Table 2-24 lists the SAS adapters that are available for Power E870 and Power E880
systems.

Table 2-24 Available SAS adapters

Featur
e Code

5901 57B3 PCIe Dual-x4 SAS Adapter I/O drawer AIX, IBM i,

5913 57B5 CIe2 1.8GB Cache RAID SAS Adapter Tri-port

ESA3 57BB PCIe2 1.8GB Cache RAID SAS Adapter

CCIN Description Placement OS support

I/O drawer AIX, Linux

Accelerator

CCIN Description Placement OS support

Linux

I/O drawer AIX, IBM i,

6Gb

I/O drawer AIX, IBM i,

Tri-port 6Gb CR

Linux

Linux

EJ0J 57B4 PCIe3 RAID SAS Adapter Quad-port 6Gb x8 I/O drawer AIX, IBM i,

72 IBM Power Systems E870 and E880 Technical Overview and Introduction

Linux

Draft Document for Review October 14, 2014 10:19 am 5137ch02.fm

Featur
e Code

EJ0L 57CE PCIe3 12GB Cache RAID SAS Adapter

EJ0M 57B4 PCIe3 LP RAID SAS ADAPTER CEC AIX, IBM i,

EJ10 57B4 PCIe3 SAS Tape/DVD Adapter Quad-port 6Gb x8I/O drawer AIX, IBM i,

EJ11 57B4 PCIe3 LP SAS Tape/DVD Adapter Quad-port

CCIN Description Placement OS support

Quad-port 6Gb x8

6Gb x8

2.7.5 Fibre Channel adapter

The systems support direct or SAN connection to devices that use Fibre Channel adapters.
Table 2-25 summarizes the available Fibre Channel adapters, which all have LC connectors.

If you are attaching a device or switch with an SC type fiber connector, then an
LC-SC 50 Micron Fiber Converter Cable (#2456) or an LC-SC 62.5 Micron Fiber Converter
Cable (#2459) is required.

Table 2-25 Available Fibre Channel adapters

Feature
Code

CCIN Description Placement OS support

I/O drawer AIX, IBM i,

Linux

Linux

Linux

CEC AIX, IBM i,

Linux

5273 577D PCIe LP 8Gb 2-Port Fibre Channel Adapter CEC AIX, IBM i,

Linux

5729 5729 PCIe2 8Gb 4-port Fibre Channel Adapter I/O drawer AIX, Linux

5735 577D Gigabit PCI Express Dual Port Fibre Channel

Adapter

5774 5774 4 Gigabit PCI Express Dual Port Fibre

Channel Adapter

EN0A 577F PCIe2 16Gb 2-port Fibre Channel Adapter I/O drawer AIX, Linux

EN0B 577F PCIe2 LP 16Gb 2-port Fibre Channel Adapter CEC AIX, IBM i,

EN0Y EN0Y PCIe2 LP 8Gb 4-port Fibre Channel Adapter CEC AIX, Linux

I/O drawer AIX, IBM i,

Linux

I/O drawer AIX, IBM i,

Linux

Linux

Chapter 2. Architecture and technical overview 73

5137ch02.fm Draft Document for Review October 14, 2014 10:19 am

2.7.6 Fibre Channel over Ethernet

Fibre Channel over Ethernet (FCoE) allows for the convergence of Fibre Channel and
Ethernet traffic onto a single adapter and a converged fabric.

Figure 2-22 compares existing Fibre Channel and network connections and FCoE
connections.

Figure 2-22 Comparison between existing Fibre Channel and network connections and FCoE
connections

Table 2-26 lists the available FCoE adapters. They are high-performance Converged Network
Adapters (CNAs) using SR optics. Each port can simultaneously provide network interface
card (NIC) traffic and Fibre Channel functions.

Table 2-26 Available FCoE adapters

Feature
Code

EN0H 2B93 PCIe2 4-port (10Gb FCoE & 1 GbE) SR&RJ45 I/O drawer AIX, Linux

EN0J 2B93 PCIe2 LP 4-port (10Gb FCoE & 1 GbE)

EN0K 2CC1 PCIe2 4-port (10Gb FCoE & 1 GbE)

EN0L 2CC1 PCIe2 LP 4-port(10Gb FCoE & 1 GbE)

CCIN Description Placement OS support

SR&RJ45

SFP+Copper&RJ45

SFP+Copper&RJ45

For more information about FCoE, see An Introduction to Fibre Channel over Ethernet, and
Fibre Channel over Convergence Enhanced Ethernet, REDP-4493.

2.7.7 Asynchronous and USB adapters

CEC AIX, Linux

I/O drawer AIX, Linux

CEC AIX, Linux

Asynchronous PCI adapters provide connection of asynchronous EIA-232 or RS-422
devices.

Each system control unit enclosure can have one slim-line bay which can support one DVD
drive (#EU13). The #EU13 DVD is cabled to a USB PCIe adapter located in either the system
node or in a PCIe Gen3 I/O drawer.

74 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch02.fm

Table 2-27 lists the available asynchronous and USB adapters.

Table 2-27 Available asynchronous and USB adapters

Feature
Code

5785 57D2 4 Port Async EIA-232 PCIe Adapter I/O drawer AIX, Linux

EN27 2 Port Async EIA-232 PCIe Adapter I/O drawer AIX, IBM i, Linux

EC45 PCIe2 LP 4-Port USB 3.0 Adapter CEC AIX, IBM i, Linux

EC46 PCIe2 4-Port USB 3.0 Adapter I/O drawer AIX, IBM i, Linux

CCIN Description Placement OS support

2.7.8 InfiniBand host channel adapter

The InfiniBand architecture (IBA) is an industry-standard architecture for server I/O and
inter-server communication. It was developed by the InfiniBand Trade Association (IBTA) to
provide the levels of reliability, availability, performance, and scalability necessary for present
and future server systems with levels significantly better than can be achieved by using
bus-oriented I/O structures.

InfiniBand (IB) is an open set of interconnect standards and specifications. The main IB
specification is published by the InfiniBand Trade Association and is available at the following
location:

http://www.infinibandta.org/

InfiniBand is based on a switched fabric architecture of serial point-to-point links, where these
IB links can be connected to either host channel adapters (HCAs), used primarily in servers,
or to target channel adapters (TCAs), used primarily in storage subsystems.

The InfiniBand physical connection consists of multiple byte lanes. Each individual byte lane
is a four-wire, 2.5, 5.0, or 10.0 Gbps bidirectional connection. Combinations of link width and
byte-lane speed allow for overall link speeds from 2.5 Gbps to 120 Gbps. The architecture
defines a layered hardware protocol, and also a software layer to manage initialization and
the communication between devices. Each link can support multiple transport services for
reliability and multiple prioritized virtual communication channels.

Table 2-28 lists the available InfiniBand adapters.

Table 2-28 Available InfiniBand adapters

Feature
Code

5285 58E2 PCIe2 2-Port 4X IB QDR Adapter 40Gb I/O drawer AIX, Linux

CCIN Description Placement OS support

2.7.9 Cryptographic Coprocessor

The Cryptographic Coprocessor cards provide both cryptographic coprocessor and
cryptographic accelerator functions in a single card.

The IBM PCIe Cryptographic Coprocessor adapter highlights the following features:

򐂰 Integrated Dual processors that operate in parallel for higher reliability
򐂰 Supports IBM Common Cryptographic Architecture or PKCS#11 standard

Chapter 2. Architecture and technical overview 75

5137ch02.fm Draft Document for Review October 14, 2014 10:19 am

򐂰 Ability to configure adapter as coprocessor or accelerator
򐂰 Support for smart card applications using Europay, MasterCard and Visa
򐂰 Cryptographic key generation and random number generation
򐂰 PIN processing: generation, verification, translation
򐂰 Encrypt and Decrypt using AES and DES keys

See the following site for the most recent firmware and software updates:

http://www.ibm.com/security/cryptocards/

Table 2-29 lists the cryptographic adapter that is available for the server.

Table 2-29 Available cryptographic adapters

Feature
Code

CCIN Description Placement OS support

EJ28 PCIe Crypto Coprocessor Gen3 BSC

2.8 Internal Storage

The system nodes for Power E870 and Power E880 do not allow for physical storage. All
storage must be provided externally via I/O expansion drawers or SAN. At the time of writing,
the only external I/O expansion drawer that can be used in Power E870 and Power E880 is
EXP24S, attached via SAS ports to a SAS PCIe adapter installed either in a system node
drawer or in a PCIe expansion slot located in an I/O expansion drawer.

The control unit has a DVD drive which is connected to an external USB port on the rear of
the unit. In order to use the DVD drive, at least on PCIe USB adapter must be installed and
connected via an USB cable to the DVD drive.

2.8.1 DVD

There is one DVD media bay per system, located on the front of the control unit. This media
bay allows for a DVD (#EU13) to be installed on the control unit and it enables the USB port
on the rear of the control unit.

The USB port must be connect via an USB cable to a USB PCIe adapter, installed in one of
the available PCIe slots on the system nodes or I/O expansion drawers. A diagram of the DVD
connection can be seen on Figure 2-23 on page 77.

4765-001

I/O drawer AIX, IBM i,

Linux

The basic rules for DVD drives on these system are:

򐂰 Include DVD drive with #EC13
򐂰 Include the PCI USB adapter #EC45
򐂰 Include a USB cable male-male with the proper length. As a suggestion, #EBK4 is a 1.6m

cable that allows enough length for the adapters on the first two system nodes to be
connected to the USB DVD port.

This architecture allows for a more flexible infrastructure where the DVD drive is completely
independent of another components and can be freely moved between partition.

76 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch02.fm

Control Unit – Front View

DVD Drive (#EU13)

Control Unit – Rear View

USB Port

PCIe USB
Adapter (#EC45)

USB Cable 1.6m
Male-Male (#EBK4)

System Node

PCIe Expansion

Drawer

It is also important to note that several daily functions where a DVD drive was needed can
now be executed by using other methods like Virtual Media Repository on the Virtual I/O
Server or the Remote Virtual I/O Server install on HMC.

2.9 External I/O subsystems

2.9.1 PCIe Gen3 I/O expansion drawer

Figure 2-23 DVD drive physical location on control unit and suggested cabling

This section describes the PCIe Gen3 I/O expansion drawer that can be attached to the
Power E870 and Power E880.

The PCIe Gen3 I/O expansion drawer is a 4U high, PCI Gen3-based and rack mountable I/O
drawer. It offers two PCIe FanOut Modules (#EMXF) that each provide six PCIe slots.

The physical dimensions of the drawer are 444.5 mm (17.5 in.) wide by 177.8 mm (7.0 in.)
high by 736.6 mm (29.0 in.) deep for use in a 19-inch rack.

A PCIe x16 to Optical CXP converter adapter (#EJ07) and 2.0 M (#ECC6) or 10.0 M (#ECC8)
CXP 16X Active Optical cables (AOC) connect the system node to a PCIe FanOut module in
the I/O expansion drawer. One feature #ECC6 or one #ECC8 ships two AOC cables.

Chapter 2. Architecture and technical overview 77

5137ch02.fm Draft Document for Review October 14, 2014 10:19 am

Fan-out Module
6 PCIe G3 Slots
4 x8 & 2 x16

4 x8 & 2 x16

Fan-out Module
6 PCIe G3 Slots
4 x8 & 2 x16

4 x8 & 2 x16

Dual Power Cords

PCIe Optical Interface to CEC

PCIe Optical Interface to CEC

Concurrent repair and add/removal of PCIe adapter cards is done by HMC guided menus or
by operating system support utilities.

A blind swap cassette (BSC) is used to house the full high adapters which go into these slots.
The BSC is the same BSC as used with the previous generation server's
#5802/5803/5877/5873 12X attached I/O drawers.

Figure 2-24 shows the back view of the PCIe Gen3 I/O expansion drawer.

Figure 2-24 Rear view of the PCIe Gen3 I/O expansion drawer

2.9.2 PCIe Gen3 I/O expansion drawer optical cabling

I/O drawers are connected to the adapters in the system node with data transfer cables:

򐂰 2M Optical Cable Pair for PCIe3 Expansion Drawer (#ECC6)
򐂰 10M Optical Cable Pair for PCIe3 Expansion Drawer (#ECC8)

Cable lengths: Use the 2-meter cables for intra-rack installations. Use the 10-meter
cables for inter-rack installations.

Two PCIe3 Optical Cable Adapter for PCIe3 Expansion Drawer (#EJ07) are required to
connect the expansion drawer, one for each of the two PCIe3 6-slot fanout modules in the
drawer. The top port of the fan out module must be cabled to the top port of the #EJ07 port.
Likewise, the bottom two ports must be cabled together.

򐂰 Connect an active optical cable to connector T1 on the PCIe3 optical cable adapter in your

server.

78 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch02.fm

򐂰 Connect the other end of the optical cable to connector T1 on one of the PCIe3 6-slot

fanout modules in your expansion drawer.

򐂰 Connect another cable to connector T2 on the PCIe3 optical cable adapter in your server.
򐂰 Connect the other end of the cable to connector T2 on the PCIe3 6-slot fanout module in

your expansion drawer.

򐂰 Repeat above 4 steps for the other PCIe3 6-slot fanout module in the expansion drawer.

Drawer connections: Each fanout module in a PCIe3 Expansion Drawer cannot be
connected to two different systems. However it can be connected to different system nodes
in a system.

Figure 2-25 shows connector locations for the PCIe Gen3 I/O expansion drawer.

Figure 2-25 Connector locations for the PCIe Gen3 I/O expansion drawer

Chapter 2. Architecture and technical overview 79

5137ch02.fm Draft Document for Review October 14, 2014 10:19 am

T1

T2

Optical cable adapter

T1

T2

Optical cable adapter

T1

T2

Fanout module

T1

T2

Fanout module

Optical cables

Optical cables

Server I/O Expansion Drawer

Figure 2-26 shows typical optical cable connections.

Figure 2-26 Typical optical cable connection

General rules for the PCI Gen3 I/O expansion drawer configuration

The PCIe3 optical cable adapter can be in any of the PCIe adapter slots in the Power E870
and Power E880 system node. However, we recommend that you use the PCIe adapter slot
priority information while selecting slots for installing PCIe3 Optical Cable Adapter (#EJ07).

Table 2-30 shows PCIe adapter slot priorities in the Power E870 and Power E880 system.

Table 2-30 PCIe adapter slot priorities

Feature code Description Slot priorities

EJ07 PCIe3 Optical Cable Adapter for PCIe3 Expansion Drawer 2, 8, 4, 6, 1, 7, 3, 5

80 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch02.fm

Left FanOut

Right FanOut

Left FanOut

Right FanOut

Slot 6

Slot 7

Slot 5

Slot 4

Slot 3

Slot 2

Slot 1

System node

I/O Draw er

I/O Draw er

Slot 8

Figure 2-27 shows example of a single system node and two PCI Gen3 I/O expansion
drawers.

Figure 2-27 Example of a single system node and two I/O drawers

Chapter 2. Architecture and technical overview 81

5137ch02.fm Draft Document for Review October 14, 2014 10:19 am

Left FanO ut

Right FanOut

Left FanOut

Right FanOut

Slot 6

Slot 7

Slot 5

Slot 4

Slot 3

Slot 2

Slot 1

Slot 6

Slot 7

Slot 5

Slot 4

Slot 3

Slot 2

Slot 1

System node

System node

I/O Drawer

I/O Drawer

Slot 8

Slot 8

Figure 2-28 shows an example of two system nodes and two PCI Gen3 I/O expansion
drawers.

Figure 2-28 Example of two system nodes and two I/O drawers

82 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch02.fm

Left FanOut

Right FanOut

Left FanOut

Right FanOut

Left FanOut

Right FanOut

Left FanOut

Right FanOut

Slot 6

Slot 7

Slot 5

Slot 4

Slot 3

Slot 2

Slot 1

Slot 6

Slot 7

Slot 5

Slot 4

Slot 3

Slot 2

Slot 1

System node

System node

I/O Drawer

I/O Drawer

I/O Drawer

I/O Drawer

Slot 8

Slot 8

Figure 2-29 shows an example of two system node and four PCI Gen3 I/O expansion
drawers.

Figure 2-29 Example of two system nodes and four I/O drawers

2.9.3 PCIe Gen3 I/O expansion drawer SPCN cabling

There is no system power control network (SPCN) used to control and monitor the status of
power and cooling within the I/O drawer. SPCN capabilities are integrated in the optical
cables.

Chapter 2. Architecture and technical overview 83

5137ch02.fm Draft Document for Review October 14, 2014 10:19 am

2.10 External disk subsystems

This section describes the following external disk subsystems that can be attached to the
Power E870 and Power E880 system:

򐂰 EXP24S SFF Gen2-bay drawer for high-density storage (#5887)
򐂰 IBM System Storage

Note: The EXP30 Ultra SSD Drawer (#EDR1 or #5888), the EXP12S SAS Disk Drawer
(#5886), and the EXP24 SCSI Disk Drawer (#5786) are not supported on the Power E870
and Power E880 system.

2.10.1 EXP24S SFF Gen2-bay Drawer

The EXP24S SFF Gen2-bay Drawer (#5887) is an expansion drawer with twenty-four 2.5-inch
form-factor SAS bays. The EXP24S supports up to 24 hot-swap SFF-2 SAS hard disk drives
(HDDs) or solid-state drives (SSDs). It uses only 2 EIA of space in a 19-inch rack. The
EXP24S includes redundant ac power supplies and uses two power cords.

To maximize configuration flexibility and space utilization, the system node of Power E870
and Power E880 system does not have integrated SAS bays or integrated SAS controllers.
PCIe SAS adapters and the EXP24S can be used to provide direct access storage.

To further reduce possible single points of failure, EXP24S configuration rules consistent with
previous Power Systems are used. IBM i configurations require the drives to be protected
(RAID or mirroring). Protecting the drives is highly recommended, but not required for other
operating systems. All Power operating system environments that are using SAS adapters
with write cache require the cache to be protected by using pairs of adapters.

84 IBM Power Systems E870 and E880 Technical Overview and Introduction

Draft Document for Review October 14, 2014 10:19 am 5137ch02.fm

GROUP 1

GROUP 1 GROUP 2

GROUP 1 GROUP 2 GROUP 3 GROUP 4

MODE 1

MODE 2

MODE 4

1 x 24 disks

2 x 12 disks

4 x 6 disks

IBM EXP24S front view

With AIX, Linux, and VIOS, you can order the EXP24S with four sets of six bays, two sets of
12 bays, or one set of 24 bays (mode 4, 2, or 1). With IBM i, you can order the EXP24S as
one set of 24 bays (mode 1). Figure 2-30 on page 85 shows the front of the unit and the
groups of disks on each mode.

Figure 2-30 EXP24S front view with location codes and disk groups depending on its mode of
operation

Mode setting is done by IBM Manufacturing and there is no option provided to change the
mode after it is shipped. The stickers indicate whether the enclosure is set to mode 1, mode
2, or mode 4. They are attached to the lower-left shelf of the chassis (A) and the center
support between the enclosure services manager modules (B).

Chapter 2. Architecture and technical overview 85

5137ch02.fm Draft Document for Review October 14, 2014 10:19 am

Figure 2-31 on page 86 shows the mode stickers.

Figure 2-31 Mode sticker locations at the rear of the 5887 disk drive enclosure

86 IBM Power Systems E870 and E880 Technical Overview and Introduction