IBM Power Systems S822LC Technical Overview And Introduction

Front cover

IBM Power Systems S822LC
for High Performance Computing

Technical Overview and Introduction

Alexandre Caldeira
Volker Haug
Scott Vetter

Redpaper

International Technical Support Organization

IBM Power Systems S822LC for High Performance
Computing

September 2016

REDP-5405-00

Note: Before using this information and the product it supports, read the information in “Notices” on page v.

First Edition (September 2016)

This edition applies to the IBM Power Systems S822LC model 8335-GTB.

© Copyright International Business Machines Corporation 2016. All rights reserved.

Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule
Contract with IBM Corp.

Contents

Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .v

Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi

IBM Redbooks promotions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

Authors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .x

Now you can become a published author, too! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .x

Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .x

Stay connected to IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi

Chapter 1. Architecture and technical description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.1 Server features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.1.1 Minimum features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.1.2 System cooling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.2 The NVIDIA Tesla P100 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.3 Operating system support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.3.1 Ubuntu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.3.2 Additional information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.4 Operating environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

1.5 Physical package . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

1.6 System architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.7 The POWER8 processor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

1.7.1 POWER8 processor overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

1.7.2 POWER8 processor core . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

1.7.3 Simultaneous multithreading. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

1.7.4 Memory access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

1.7.5 On-chip L3 cache innovation and Intelligent Cache . . . . . . . . . . . . . . . . . . . . . . . 13

1.7.6 L4 cache and memory buffer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

1.7.7 Hardware transactional memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

1.7.8 Coherent Accelerator Processor Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

1.7.9 NVLink . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

1.8 Memory subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

1.8.1 Memory riser cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

1.8.2 Memory placement rules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

1.8.3 Memory bandwidth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

1.9 System bus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

1.10 Internal I/O subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

1.11 Slot configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

1.12 System ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

1.13 PCI adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

1.13.1 PCI Express . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

1.13.2 LAN adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

1.13.3 Compute Intensive Accelerator. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

1.13.4 Fibre Channel adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

1.13.5 CAPI enabled Infiniband adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

1.14 Internal storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

1.14.1 Disk and media features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

1.15 External I/O subsystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

© Copyright IBM Corp. 2016. All rights reserved. iii

1.16 IBM System Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

1.17 Java. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

Chapter 2. Management, Reliability, Availability, and Serviceability . . . . . . . . . . . . . . 31

2.1 Main management components overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

2.2 Service processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

2.2.1 Open Power Abstraction Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

2.2.2 Intelligent Platform Management Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

2.3 Reliability, availability, and serviceability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

2.3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

2.3.2 IBM terminology versus x86 terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

2.3.3 Error handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

2.3.4 Serviceability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

2.3.5 Manageability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

Appendix A. Server racks and energy management . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

IBM server racks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

Water cooling option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

IBM 7014 Model S25 rack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

IBM 7014 Model T00 rack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

IBM 7014 Model T42 rack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

IBM 42U SlimRack 7965-94Y . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

The AC power distribution unit and rack content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

Rack-mounting rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

Useful rack additions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

OEM racks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

Energy management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

IBM EnergyScale technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

On Chip Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

Energy consumption estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

Other publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

Online resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

Help from IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

iv IBM Power Systems S822LC for High Performance Computing

Notices

This information was developed for products and services offered in the U.S.A.

IBM may not offer the products, services, or features discussed in this document in other countries. Consult
your local IBM representative for information on the products and services currently available in your area. Any
reference to an IBM product, program, or service is not intended to state or imply that only that IBM product,
program, or service may be used. Any functionally equivalent product, program, or service that does not
infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to
evaluate and verify the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in this document. The
furnishing of this document does not grant you any license to these patents. You can send license inquiries, in
writing, to:

IBM Director of Licensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785 U.S.A.

The following paragraph does not apply to the United Kingdom or any other country where such
provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION

PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR
IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT,
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of
express or implied warranties in certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodically made
to the information herein; these changes will be incorporated in new editions of the publication. IBM may make
improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time
without notice.

Any references in this information to non-IBM websites are provided for convenience only and do not in any
manner serve as an endorsement of those websites. The materials at those websites are not part of the
materials for this IBM product and use of those websites is at your own risk.

IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring
any obligation to you.

Any performance data contained herein was determined in a controlled environment. Therefore, the results
obtained in other operating environments may vary significantly. Some measurements may have been made
on development-level systems and there is no guarantee that these measurements will be the same on
generally available systems. Furthermore, some measurements may have been estimated through
extrapolation. Actual results may vary. Users of this document should verify the applicable data for their
specific environment.

Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products and cannot confirm the
accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the
capabilities of non-IBM products should be addressed to the suppliers of those products.

This information contains examples of data and reports used in daily business operations. To illustrate them
as completely as possible, the examples include the names of individuals, companies, brands, and products.
All of these names are fictitious and any similarity to the names and addresses used by an actual business
enterprise is entirely coincidental.

COPYRIGHT LICENSE:

This information contains sample application programs in source language, which illustrate programming
techniques on various operating platforms. You may copy, modify, and distribute these sample programs in
any form without payment to IBM, for the purposes of developing, using, marketing or distributing application
programs conforming to the application programming interface for the operating platform for which the sample
programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore,
cannot guarantee or imply reliability, serviceability, or function of these programs.

© Copyright IBM Corp. 2016. All rights reserved. v

Trademarks

IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines
Corporation in the United States, other countries, or both. These and other IBM trademarked terms are
marked on their first occurrence in this information with the appropriate symbol (® or ™), indicating US
registered or common law trademarks owned by IBM at the time this information was published. Such
trademarks may also be registered or common law trademarks in other countries. A current list of IBM
trademarks is available on the Web at http://www.ibm.com/legal/copytrade.shtml

The following terms are trademarks of the International Business Machines Corporation in the United States,
other countries, or both:

AIX®
DS8000®
Easy Tier®
EnergyScale™
IBM®
IBM FlashSystem®
IBM z™
POWER®

POWER Hypervisor™
Power Systems™
POWER8®
PowerHA®
PowerLinux™
PowerPC®
PowerVM®
Real-time Compression™

Redbooks®
Redpaper™
Redbooks (logo) ®
Storwize®
System Storage®
XIV®

The following terms are trademarks of other companies:

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

Java, and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its
affiliates.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Other company, product, or service names may be trademarks or service marks of others.

vi IBM Power Systems S822LC for High Performance Computing

IBM REDBOOKS PROMOTIONS

Find and read thousands of
IBM Redbooks publications

Search, bookmark, save and organize favorites

Get up-to-the-minute Redbooks news and announcements

Link to the latest Redbooks blogs and videos

Download

Now

Get the latest version of the Redbooks Mobile App

iOS

Android

Place a Sponsorship Promotion in an IBM
Redbooks publication, featuring your business
or solution with a link to your web site.

Qualified IBM Business Partners may place a full page
promotion in the most popular Redbooks publications.
Imagine the power of being seen by users who download
millions of Redbooks publications each year!

®

®

Promote your business
in an IBM Redbooks
publication

ibm.com/Redbooks

About Redbooks Business Partner Programs

IBM Redbooks promotions

THIS PAGE INTENTIONALLY LEFT BLANK

Preface

This IBM® Redpaper™ publication is a comprehensive guide that covers the IBM Power
Systems™ S822LC 8335-GTB server designed for impressive High Performance Computing
applications that support the Linux operating system (OS) as well as High Performance Data
Analytics, the enterprise datacenter, and accelerated cloud deployments.

The objective of this paper is to introduce the major innovative Power S822LC features and
their relevant functions:

򐂰 Powerful POWER8® processors that offer 16 cores at 3.259 GHz or 3.857 GHz turbo

򐂰 A 19-inch rack-mount 2U configuration
򐂰 NVIDIA NVLink technology for exceptional processor to accelerator intercommunication
򐂰 Four SXM2 form factor connectors for the NVIDIA Tesla P100 GPU

This publication is for professionals who want to acquire a better understanding of IBM Power
Systems products. The intended audience includes the following roles:

򐂰 Clients
򐂰 Sales and marketing professionals
򐂰 Technical support professionals

performance or 20 cores at 2.860 GHz or 3.492 GHz turbo

򐂰 IBM Business Partners
򐂰 Independent software vendors

This paper expands the set of IBM Power Systems documentation by providing a desktop
reference that offers a detailed technical description of the Power S822LC server.

This paper does not replace the latest marketing materials and configuration tools. It is
intended as an additional source of information that, together with existing sources, can be
used to enhance your knowledge of IBM server solutions.

© Copyright IBM Corp. 2016. All rights reserved. ix

Authors

This paper was produced by a team of specialists from around the world working at the
International Technical Support Organization, Austin Center.

Alexandre Caldeira is a Certified IT Specialist and is the Product Manager for Power
Systems Latin America. He holds a degree in Computer Science from the Universidade
Estadual Paulista (UNESP) and an MBA in Marketing. His major areas of focus are
competition, sales, marketing and technical sales support. Alexandre has more than 16 years
of experience working on IBM Systems Solutions and has worked also as an IBM Business
Partner on Power Systems hardware, AIX, and IBM PowerVM® virtualization products.

Volker Haug is an Executive IT Specialist & Open Group Distinguished IT Specialist within
IBM Systems in Germany supporting Power Systems clients and Business Partners. He
holds a Diploma degree in Business Management from the University of Applied Studies in
Stuttgart. His career includes more than 29 years of experience with Power Systems, AIX,
and PowerVM virtualization. He has written several IBM Redbooks® publications about
Power Systems and PowerVM. Volker is an IBM POWER8® Champion and a member of the
German Technical Expert Council, which is an affiliate of the IBM Academy of Technology.

The project that produced this publication was managed by:

Scott Vetter

Executive Project Manager, PMP

Thanks to the following people for their contributions to this project:

George Ahrens, Nick Bofferding, Charlie Burns, Sertac Cakici, Dan Crowell,
Daniel Henderson, Yesenia Jimenez, Ann Lund, Benjamin Mashak, Chris Mann,
Michael J Mueller, Kanisha Patel, Matt Spinler, Jeff Stuecheli, Uma Yadlapati, and
Maury Zipse.

Now you can become a published author, too!

Here’s an opportunity to spotlight your skills, grow your career, and become a published
author—all at the same time! Join an ITSO residency project and help write a book in your
area of expertise, while honing your experience using leading-edge technologies. Your efforts
will help to increase product acceptance and customer satisfaction, as you expand your
network of technical contacts and relationships. Residencies run from two to six weeks in
length, and you can participate either in person or as a remote resident working from your
home base.

Find out more about the residency program, browse the residency index, and apply online at:

ibm.com/redbooks/residencies.html

Comments welcome

Your comments are important to us!

x IBM Power Systems S822LC for High Performance Computing

We want our papers to be as helpful as possible. Send us your comments about this paper or
other IBM Redbooks publications in one of the following ways:

򐂰 Use the online Contact us review Redbooks form found at:

ibm.com/redbooks

򐂰 Send your comments in an email to:

[e-mail]

򐂰 Mail your comments to:

IBM Corporation, International Technical Support Organization
Dept. HYTD Mail Station P099
2455 South Road
Poughkeepsie, NY 12601-5400

Stay connected to IBM Redbooks

򐂰 Find us on Facebook:

http://www.facebook.com/IBMRedbooks

򐂰 Follow us on Twitter:

http://twitter.com/ibmredbooks

򐂰 Look for us on LinkedIn:

http://www.linkedin.com/groups?home=&gid=2130806

򐂰 Explore new Redbooks publications, residencies, and workshops with the IBM Redbooks

weekly newsletter:

https://www.redbooks.ibm.com/Redbooks.nsf/subscribe?OpenForm

򐂰 Stay current on recent Redbooks publications with RSS Feeds:

http://www.redbooks.ibm.com/rss.html

Preface xi

xii IBM Power Systems S822LC for High Performance Computing

Chapter 1. Architecture and technical

1

description

The IBM Power System S822LC for High Performance Computing (8335-GTB) server, the
first Power System offering with NVIDIA NVLink Technology, removes GPU computing
bottlenecks by employing the high-bandwith and low-latency NVLink interface from CPU to
GPU and GPU to GPU. This unlocks new performance and new applications for accelerated
computing.

Power System LC servers are a product of a co-design with OpenPOWER Foundation
ecosystem members. Power S822LC for High Performance Computing innovation partners
include IBM, NVIDIA, Mellanox, Canonical, Wistron, and more.

The Power S822LC server offers a modular design to scale from single racks to hundreds,
simplicity of ordering, and a strong innovation roadmap for GPUs.

The IBM Power System S822LC for High Performance Computing (8335-GTB) server, the
first Power System offering with NVIDIA NVLink Technology, removes GPU computing
bottlenecks by employing the high-bandwith and low-latency NVLink interface from CPU to
GPU and GPU to GPU. This unlocks new performance and new applications for accelerated
computing

The IBM Power S822LC for High Performance Computing (8335-GTB) server offers two
processor sockets for a total of 16 cores at 3.259 GHz (3.857 GHz turbo) or 20 cores at 2.860
GHz (3.492 GHz turbo) in a 19-inch rack-mount, 2U (EIA units) drawer configuration. All the
cores are activated.

The server provides eight memory daughter cards with 16 GB (4x4 GB), 32 GB (4x8 GB),
64 GB (4x16 GB), and 128 GB (4x32 GB), allowing for a maximum system memory of
1024 GB.

© Copyright IBM Corp. 2016. All rights reserved. 1

Figure 1-1 shows the front view of a Power S822LC server.

POWER8 Processor (2x)

Operator Interface

•1 USB 3.0

• Green, Amber, Blue LED’s

Cooling Fans (x4)

• Coun ter- Rotat ing

• Hot s wap

PCIe slot (1x)

• PCIe Gen3 x8 (CAPI)

PCIe slot (2x)

• PCIe Gen3 x16 (CAPI)

• PCIe Gen3 x8

NIVIDIA P100 GPU (4x)

• PCIe Gen3 (CAPI)

• 300W Capable

• PCIe riser cards

Power Supplies (2x)

• 1,300 W

• Hot Swap

Memory riser cards (8x)

• 1 Buffer Chip per Riser

• 4 DIMMs per Riser

• 32 DIMMs total

Front Bezel removed

Internal Storage

• 2 SFF-4 bays

• HHD or SSD

Figure 1-1 Front view of the Power S822LC server

1.1 Server features

The server chassis of the Power S822LC server contains two processor modules attached
directly to the board. Each POWER8 processor module is either 8-core or 10-core and has a
64-bit architecture, up to 512 KB of L2 cache per core, and up to 8 MB of L3 cache per core.
The clock speed of each processor available varies based on the model of the server that is
used.

The Power S822LC computing server provides eight DIMM memory slots. Memory features
that are supported are 16 GB (#EM55), 32 GB (#EM56), 64 GB (#EM57), and 128 GB
(#EM58), allowing for a maximum of 1024 GB DDR4 system memory.

The physical locations of the main server components are shown in Figure 1-2.

Figure 1-2 Location of server main components

The servers support four SXM2 form factor connectors for NVIDIA Tesla P100 GPU (#EC4C,
#EC4D, or #EC4F) only. Optional water cooling is available.

This summary describes the standard features of the Power S822LC model 8355-GTA
server:

򐂰 19” rack-mount (2U) chassis

2 IBM Power Systems S822LC for High Performance Computing

򐂰 Two POWER8 processor modules:

– 8-core 3.3259 GHz processor module
– 10-core 2.860 GHz processor module

Up to 1024 GB of 1333 MHz DDR4 ECC memory

򐂰 Two SFF bays for two HDDs or two SSDs that supports:

– Two 1 TB 7200 RPM NL SATA disk drives (#ELD0)
– Two 2 TB 7200 RPM NL SATA disk drives (#ES6A)
– Two 480 GB SATA SSDs (#ELS5)
– Two 960 GB SATA SSDs (#ELS6)
– Two 1.92 TB SATA SSDs (#ELSZ)
– Two 3.84 TB SATA SSDs (#ELU0)

򐂰 Integrated SATA controller
򐂰 Three PCIe Gen 3 slots:

– One PCIe x8 Gen3 Low Profile slot, CAPI enabled
– Two PCIe x16 Gen3 Low Profile slot, CAPI enabled

򐂰 Four SXM2 form factor connectors for NVIDIA Tesla P100 GPU (#EC4C, #EC4D, or

#EC4F) only

򐂰 Integrated features:

– EnergyScale™ technology
– Hot-swap and redundant cooling
– One front USB 2.0 port for general usage
– One rear USB 3.0 port for general usage
– One system port with RJ45 connector

򐂰 Two power supplies

1.1.1 Minimum features

The minimum Power S822LC model 8355-GTB server initial order must include:

򐂰 Two processor modules with at least 16 CPUs
򐂰 128 GB of memory (eight 16 GB memory DIMMs)
򐂰 Two #EC4C Compute Intensive Accelerator - NVIDIA GP100
򐂰 Two power supplies and power cords
򐂰 An OS indicator
򐂰 A rack integration indicator
򐂰 A Language Group Specify

Linux is the supported OS. The Integrated 1Gb Ethernet port can be used as the base LAN
port.

1.1.2 System cooling

Air or water cooling depends on the GPU that is installed. See 1.13.3, “Compute Intensive
Accelerator” on page 26 for a list of GPUs available.

Feature code #ER2D is the water cooling indicator for the 8335-GTB.

Note: If #ER2D is ordered, you must order #EJTX fixed rail kit. Ordering #ER2D with
#EJTY slide rails is not supported.

Chapter 1. Architecture and technical description 3

1.2 The NVIDIA Tesla P100

NVIDIA’s new NVIDIA Tesla P100 accelerator (see Figure 1-3) takes GPU computing to the
next level. This section discusses the Tesla P100 accelerator.

Figure 1-3 NVIDIA Tesla P100 accelerator

With a 15.3 billion transistor GPU, a new high performance interconnect that greatly
accelerates GPU peer-to-peer and GPU-to-CPU communications, new technologies to
simplify GPU programming, and exceptional power efficiency, Tesla P100 is not only the most
powerful, but also the most architecturally complex GPU accelerator architecture ever built.

Key features of Tesla P100 include:
򐂰 Extreme performance

Powering HPC, Deep Learning, and many more GPU Computing areas

򐂰 NVLink

NVIDIA’s new high speed, high bandwidth interconnect for maximum application
scalability

򐂰 HBM2

Fast, high capacity, extremely efficient CoWoS (Chip-on-Wafer-on-Substrate) stacked
memory architecture

򐂰 Unified Memory, Compute Preemption, and New AI Algorithms

Significantly improved programming model and advanced AI software optimized for the
Pascal architecture;

򐂰 16nm FinFET

Enables more features, higher performance, and improved power efficiency

4 IBM Power Systems S822LC for High Performance Computing

The Tesla P100 was built to deliver exceptional performance for the most demanding
compute applications, delivering:

򐂰 5.3 TFLOPS of double precision floating point (FP64) performance
򐂰 10.6 TFLOPS of single precision (FP32) performance
򐂰 21.2 TFLOPS of half-precision (FP16) performance

In addition to the numerous areas of high performance computing that NVIDIA GPUs have
accelerated for a number of years, most recently Deep Learning has become a very important
area of focus for GPU acceleration. NVIDIA GPUs are now at the forefront of deep neural
networks (DNNs) and artificial intelligence (AI). They are accelerating DNNs in various
applications by a factor of 10x to 20x compared to CPUs, and reducing training times from
weeks to days. In the past three years, NVIDIA GPU-based computing platforms have helped
speed up Deep Learning network training times by a factor of fifty. In the past two years, the
number of companies NVIDIA collaborates with on Deep Learning has jumped nearly 35x to
over 3,400 companies.

New innovations in the Pascal architecture, including native 16-bit floating point (FP)
precision, allow GP100 to deliver great speedups for many Deep Learning algorithms. These
algorithms do not require high levels of floating-point precision, but they gain large benefits
from the additional computational power FP16 affords, and the reduced storage requirements
for 16-bit datatypes.

For more detailed information on the NVIDA Tesla P100, see:

https://devblogs.nvidia.com/parallelforall/inside-pascal/

1.3 Operating system support

The Power S822LC (8335-GTB) server supports Linux, which provides a UNIX like
implementation across many computer architectures.

For more information about the software that is available on Power Systems, see the Linux on
Power Systems website:

http://www.ibm.com/systems/power/software/linux/index.html

The Linux operating system is an open source, cross-platform OS. It is supported on every
Power Systems server IBM sells. Linux on Power Systems is the only Linux infrastructure that
offers both scale-out and scale-up choices.

1.3.1 Ubuntu

Ubuntu Server 16.04, at the time of writing, is the supported OS for the S822LC.

For more information about Ubuntu Server for Ubuntu for POWER8, see the following
website:

http://www.ubuntu.com/download/server/power8

1.3.2 Additional information

For more information about the IBM PowerLinux™ Community, see the following website:

https://www.ibm.com/developerworks/group/tpl

Chapter 1. Architecture and technical description 5

For more information about the features and external devices that are supported by Linux,
see the following website:

http://www.ibm.com/systems/power/software/linux/index.html

1.4 Operating environment

Table 1-1 provides the operating environment specifications for the Power S822LC servers.

Table 1-1 Operating environment for Power S822LC servers

Power S822LC server operating environment

Description Operating Non-operating

Temperature Allowable: 5 - 40 degrees Ca

(41 - 104 degrees F)
Recommended: 18 - 27
degrees C
(64 - 80 degrees F)

Relative humidity 8 - 80% 8 - 80%

Maximum dew point 24 degrees C (75 degrees F) 27 degrees C (80 degrees F)

Operating voltage 200 - 240 V AC N/A

Operating frequency 50 - 60 Hz +/- 3 Hz N/A

Power consumption 2550 watts maximum N/A

Power source loading 2.6 kVA maximum N/A

Thermal output 8703 BTU/hr maximum N/A

Maximum altitude 3,050 m

(10,000 ft)

Noise level and sound power 7.6/6.7 bels operating/ idling N/A

a. Heavy workloads might see some performance degradation above 35 degrees C if internal

temperatures trigger a CPU clock reduction.

1 - 60 degrees C
(34 - 140 degrees F)

N/A

Tip: The maximum measured value is expected from a fully populated server under an
intensive workload. The maximum measured value also accounts for component tolerance
and operating conditions that are not ideal. Power consumption and heat load vary greatly
by server configuration and usage. Use the IBM Systems Energy Estimator to obtain a
heat output estimate that is based on a specific configuration, which is available at the
following website:

http://www-912.ibm.com/see/EnergyEstimator

1.5 Physical package

Table 1-2 on page 7 shows the physical dimensions of the Power S822LC chassis. The
servers are available only in a rack-mounted form factor and take 2U (2 EIA units) of rack
space.

6 IBM Power Systems S822LC for High Performance Computing

Table 1-2 Physical dimensions for the Power S822LC servers

Dimension Power S822LC (8335-GTB) server

Width 441.5 mm (17.4 in.)

Depth 822 mm (32.4 in.)

Height 86 mm (3.4 in.)

Weight (maximum configuration) 30 kg (65 lbs.)

1.6 System architecture

This section describes the overall system architecture for the Power S822LC computing
servers. The bandwidths that are provided throughout the section are theoretical maximums
that are used for reference.

The speeds that are shown are at an individual component level. Multiple components and
application implementation are key to achieving the preferred performance. Always do the
performance sizing at the application workload environment level and evaluate performance
by using real-world performance measurements and production workloads.

The Power S822LC server is a two single chip module (SCM) system. Each SCM is attached
to four memory riser cards that have buffer chips for the L4 Cache and four memory RDIMM
slots. The server has a maximum capacity of 32 memory DIMMs when all the memory riser
cards are populated, which allows for up to 1024 GB of memory.

The servers have a total of three PCIe Gen3 slots with all of these slots being CAPI-capable.
The system has sockets for four GPUs each 300 W capable.

An integrated SATA controller is fed through a dedicated PCI bug on the main system board
and allows for up to two SATA HDDs or SSDs to be installed. This bus also drives the
integrated Ethernet and USB port.

Chapter 1. Architecture and technical description 7

B
u
f

f

e
r
C
a
c
h
e

L
4

R
D
I

M
M

R
D
I

M
M

R
D
I

M
M

R
D
I

M
M

B
u
f

f

e
r
C
a
c
h
e

L
4

R
D
I

M
M

R
D
I

M
M

R
D
I

M
M

R
D
I

M
M

B
u
f

f

e
r
C
a
c
h
e

L
4

R
D
I

M
M

R
D
I

M
M

R
D
I

M
M

R
D
I

M
M

B
u
f

f

e
r
C
a
c
h
e

L
4

R
D
I

M
M

R
D
I

M
M

R
D
I

M
M

R
D
I

M
M

B
u
f

f

e
r
C
a
c
h
e

L
4

R
D
I

M
M

R
D
I

M
M

R
D
I

M
M

R
D
I

M
M

B
u
f

f

e
r
C
a
c
h
e

L
4

R
D
I

M
M

R
D
I

M
M

R
D
I

M
M

R
D
I

M
M

B
u
f

f

e
r
C

a
c
h
e

L
4

R
D
I

M
M

R
D
I

M
M

R
D
I

M
M

R
D
I

M
M

B
u
f

f

e
r
C
a
c
h
e

L
4

R
D
I

M
M

R
D
I

M
M

R
D
I

M
M

R
D
I

M
M

P
H
B

0

P
H
B

1

S
M
P

A
B

u
s

M

e

m

o

r

y

C

o

n

t

r

o

l

l

e

r

P
H
B

0

P
H
B

1

P
C
I

e
G
e

n
3

x
8

­C
A
P
I
s
u
p
p
o
r

t

P
C
I

e
G
e

n
3

x
1
6

­C
A
P
I
s
u
p
p
o
r

t

P
C
I

e
G
e

n
3

x
1
6

­C
A
P
I
s
u
p
p
o
r

t

SATA

Controller

SFF-4

HDD/

SSD

USB 3.0

Front

USB 3.0

Rear

VGA

Management

1 Gbps

Ethernet

System Planar

12.8 GBps

28.8 GBps

x
8

x
1
6

x
8

x
1
6

POWER8

SCM0

POWER8

SCM1

SFF-4

HDD/

SSD

PLX

M

e

m

o

r

y

C

o

n

t

r

o

l

l

e

r

M

e

m

o

r

y

C

o

n

t

r

o

l

l

e

r

M

e

m

o

r

y

C

o

n

t

r

o

l

l

e

r

S
M
P

A
B

u
s

N
V
L
i

n
k

N
V
L
i

n
k

N
V
L

i

n
k

N
V
L
i

n
k

N
V
I

D
I

A
P

1
0
0

S
X
M

2
.

0

N
V
I

D
I

A
P

1
0
0

S
X
M

2
.

0

N
V
I

D
I

A
P

1
0
0

S
X
M

2
.

0

N
V
I

D
I

A
P

1
0
0

S
X
M

2
.

0

S
M
P

A
B

u
s

S
M
P

A
B

u
s

Figure 1-4 shows the logical system diagram for the Power S822LC servers.

Figure 1-4 Power S822LC server logical system diagram

8 IBM Power Systems S822LC for High Performance Computing

1.7 The POWER8 processor

This section introduces the latest processor in the Power Systems product family and
describes its main characteristics and features in general.

The POWER8 processor in the S822LC for High Performance Computing is unique to the
8335-GTB. Engineers removed the A-Bus interface along with SMP over PCI support to make
room for the NVLink interface. The resulting chip grows slightly from 649 mm

1.7.1 POWER8 processor overview

2

to 659 mm2.

The POWER8 processor is manufactured by using the IBM 22 nm Silicon-On-Insulator (SOI)
technology. Each chip is 649 mm
the chip contains up to 12 cores, two memory controllers, Peripheral Component Interconnect
Express (PCIe) Gen3 I/O controllers, and an interconnection system that connects all
components within the chip. Each core has 512 KB of L2 cache, and all cores share 96 MB of
L3 embedded DRAM (eDRAM). The interconnect also extends through module and system
board technology to other POWER8 processors in addition to DDR4 memory and various I/O
devices.

POWER8 processor-based systems use memory buffer chips to interface between the
POWER8 processor and DDR4 memory. Each buffer chip also includes an L4 cache to
reduce the latency of local memory accesses.

2

and contains 4.2 billion transistors. As shown in Figure 1-5,

Figure 1-5 The POWER8 processor chip

Chapter 1. Architecture and technical description 9

Here are additional features that can augment the performance of the POWER8 processor:
򐂰 Support for DDR4 memory through memory buffer chips that offload the memory support

from the POWER8 memory controller.

򐂰 An L4 cache within the memory buffer chip that reduces the memory latency for local

access to memory behind the buffer chip; the operation of the L4 cache is not apparent to
applications running on the POWER8 processor. Up to 128 MB of L4 cache can be
available for each POWER8 processor.

򐂰 Hardware transactional memory.
򐂰 On-chip accelerators, including on-chip encryption, compression, and random number

generation accelerators.

򐂰 CAPI, which allows accelerators that are plugged into a PCIe slot to access the processor

bus by using a low latency, high-speed protocol interface.

򐂰 Adaptive power management.

Table 1-3 summarizes the technology characteristics of the POWER8 processor.

Table 1-3 Summary of POWER8 processor technology

Technology 8335-GTB POWER8 processor

Die size 659 mm

Fabrication technology 򐂰 22 nm lithography

򐂰 Copper interconnect
򐂰 SOI
򐂰 eDRAM

2

Maximum processor cores 12

Maximum execution threads core/chip 8/96

Maximum L2 cache core/chip 512 KB/6 MB

Maximum On-chip L3 cache core/chip 8 MB/96 MB

Maximum L4 cache per chip 128 MB

Maximum memory controllers 2

SMP design-point 16 sockets with POWER8 processors

Compatibility Specific to the 8335-GTB

10 IBM Power Systems S822LC for High Performance Computing