Intel Stratix 10 MX HBM2 IP User Manual

Intel® Stratix® 10 MX HBM2 IP User Guide

Updated for Intel® Quartus® Prime Design Suite: 17.1

Send Feedback

UG-20031 | December 2017

1 Introduction to High Bandwidth Memory

High Bandwidth Memory (HBM) is a JEDEC specification (JESD-235) for a wide, high bandwidth memory device. The next generation of High Bandwidth Memory, HBM2, is defined in JEDEC specification JESD-235A. The HBM2 implementation in Intel Stratix® 10 MX devices complies to JESD-235A.

The High Bandwidth Memory DRAM is tightly coupled to the host die with a distributed interface. The interface is divided into independent channels, each completely independent of one another. Each channel interface maintains a 128-bit data bus, operating at DDR data rates.

1.1 HBM2 in Intel Stratix 10 MX Devices

Intel Stratix 10 MX incorporates a high-performance FPGA fabric along with a HBM2 DRAM in a single package. Intel Stratix 10 MX devices support up to a maximum of two HBM2 interfaces.

Intel Stratix 10 MX incorporates Intel’s Embedded Multi-Die Interconnect Bridge (EMIB) technology to implement a silicon bridge between HBM2 DRAM memory and the Universal Interface Block Subsystem (UIBSS), which contains the HBM2 controller (HBMC), physical-layer interface (PHY), and I/O ports to interface to the HBM2 stack.

As illustrated below, each Intel Stratix 10 MX device contains a single universal interface bus per HBM2 interface, supporting 8 independent channels.

The user interface to the HBM2 controller is maintained through the AIX4 protocol. Sixteen AXI interfaces are available in the user interface from each HBM2 controller, with one AXI interface available per HBM2 Pseudo Channel. HBM2 DRAM density of 4GB and 8GB are supported.

Figure 1. Intel Stratix 10 MX Device with UIB, EMIB, and HBM2 DRAM

Intel Corporation. All rights reserved. Intel, the Intel logo, Altera, Arria, Cyclone, Enpirion, MAX, Nios, Quartus and Stratix words and logos are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. Intel warrants performance of its FPGA and semiconductor products to current specifications in accordance with Intel's standard warranty, but reserves the right to make changes to any products and services at any time without notice. Intel assumes no responsibility or liability arising out of the application or use of any information, product, or service described herein except as expressly agreed to in writing by Intel. Intel customers are advised to obtain the latest version of device specifications before relying on any published information and before placing orders for products or services. *Other names and brands may be claimed as the property of others.

ISO 9001:2008 Registered

1 Introduction to High Bandwidth Memory

1.2 HBM2 DRAM Structure

The HBM DRAM is optimized for high-bandwidth operation to a stack of multiple DRAM devices across several independent interfaces called channels. Each DRAM stack supports up to eight channels.

The following figure shows an example stack containing four DRAM dies, each die supporting two channels. Each die contributes additional capacity and additional channels to the stack, up to a maximum of eight channels per stack. Each channel provides access to an independent set of DRAM banks. Requests from one channel may not access data attached to a different channel.

Figure 2. High Bandwidth Memory Stack of Four DRAM Dies

UG-20031 | December 2017

1.3 Intel Stratix 10 MX HBM2 Features

Intel Stratix 10 MX FPGAs offer the following HBM2 features.

• Supports one to eight HBM2 channels per HBM2 interface in the Pseudo Channel mode.

• Each HBM2 channel supports a 128-bit DDR data bus, with optional ECC support.

• Pseudo Channel mode divides a channel into two individual 64-bit data interfaces per channel. The Pseudo Channels share the same Address and Command bus, but decodes and executes commands individually.

•

Data referenced to strobes RDQS_t / RDQS_c and WDQS_t / WDQS_c, one strobe pair per 32 DQs.

•

Differential clock inputs (CK_t / CK_c). Unterminated data/address/cmd/clk interfaces.

Intel® Stratix® 10 MX HBM2 IP User Guide 4

1 Introduction to High Bandwidth Memory

UG-20031 | December 2017

•

DDR commands entered on each positive CK_t and CK_c edge. Row Activate commands require two memory cycles; all other command are single-cycle commands.

• Supports command, write data and read data parity.

• Support for bank grouping.

• Support for data bus inversion.

• Data mask for masking write data per byte. (Not available with ECC.)

• I/O voltage of 1.2V and DRAM core voltage of 1.2V.

1.4 Intel Stratix 10 MX HBM2 Controller Features

Intel Stratix 10 MX FPGAs offer the following controller features.

• User applications communicate with the HBMC using the AXI4 Protocol.

• There is one AXI4 interface per HBM2 Pseudo Channel. Each HBM2 interface supports a maximum of sixteen AXI4 interfaces to the sixteen Pseudo Channels.

• The full-rate user interface can operate at a frequency lower than the HBM2 interface frequency For information on supported clock frequencies, refer to Intel

Stratix 10 MX HBM2 Supported Frequencies in Intel Stratix 10 MX HBM2 IP Controller Interface Signals.

• The controller offers 32B and 64B access granularity supporting burst length 4 (BL

4) and pseudo-BL 8 (two back to back BL4).

• The controller offers out-of-order command scheduling and read data reordering.

• The controller supports a user-initiated refresh command (enabled through the side band Advanced Peripheral Bus (APB) interface).

• The controller supports data mask or error correction code (ECC). When you do not use data mask or ECC, you may use those bits as additional data bits.

2 Intel Stratix 10 MX HBM2 Architecture

This chapter provides an overview of the Intel Stratix 10 MX HBM2 architecture.

2.1 Intel Stratix 10 MX HBM2 Introduction

Intel Stratix 10 MX devices use the Intel EMIB technology to interface to the HBM2 memory devices.

• The Intel Stratix 10 MX FPGAs offer up to two HBM2 interfaces.

• Each HBM2 device can have a device density of 4GB or 8GB, based on the FPGA chosen.

This system-in-package solution helps to achieve maximum bandwidth and low power consumption in a small footprint.

2.2 Intel Stratix 10 MX HBM2 Architecture

The Intel Stratix 10 MX device architecture includes the universal interface bus (UIB) subsystem (UIBSS) which contains the necessary logic to interface the FPGA core to the HBM2 DRAM.

Each UIB subsystem includes the HBM2 hardened controller and the universal interface bus, consisting of the hardened physical interface and I/O logic needed to interface to each HBM2 DRAM device. The AMBA AXI4 protocol interfaces the core logic with the universal interface bus subsystem. An optional soft logic adapter implemented in the FPGA fabric helps to efficiently interface user logic to the hardened HBM2 controller.

The following figure shows a high-level block diagram of the Intel Stratix 10 HBM2 universal interface bus subsystem. The UIB subsystem includes the following hardened logic:

• Rate-matching FIFOs that transfer logic from the user core clock to the HBM2 clock domain.

• HBM2 memory controller (HBMC).

• UIB PHY, including the UIB physical layer and I/O.

ISO 9001:2008 Registered

2 Intel Stratix 10 MX HBM2 Architecture

UG-20031 | December 2017

Figure 3. Block Diagram of Intel Stratix 10 MX HBM2 Implementation

The user core clock drives the logic highlighted in green, while the UIB clocks the logic highlighted in blue. The UIB clock also drives the HBM2 interface clock. User logic can run up to one-to-four times slower than the HBM2 interface.

Soft Logic AXI Adaptor

The HBM2 IP also includes a soft logic adaptor implemented in core logic. The soft logic adaptor gates the user valid signals (write address valid, write data

valid, and read address valid) with the corresponding pipelined ready signals

from the HBM2 controller. The soft logic adapter also temporarily stores output from the HBM2 controller (AXI write response and AXI read data channels) when the AXI ready signal is absent. You can disable the temporary storage logic if user logic is always ready to accept output from the HBM2 controller through the parameter editor when generating the HBM2 IP.

HBM2 DRAM

The HBM2 DRAM is ideal for high-bandwidth operation to multiple DRAM devices across many independent interfaces called channels. Each channel provides access to an independent set of DRAM banks. Requests from one channel cannot access data attached to another channel.

Each HBM2 channel consists of an independent command and data interface to and from the HBM2 DRAM. A channel provides access to a discrete pool of memory in the DRAM device; no channel can access the memory storage for another channel. Each channel interface provides an independent interface to a specific number of banks of DRAM of a defined page size.

The following table lists the HBM2 signals that interface to the UIB. The UIB drives the HBM2 signals and decodes the received data from the HBM2. These signals cannot be accessed through the AXI4 User Interface.

Intel® Stratix® 10 MX HBM2 IP User Guide

2 Intel Stratix 10 MX HBM2 Architecture

UG-20031 | December 2017

Table 1. Summary of Per-channel Signals

Signal Name Signal Width Notes

Data 128 128 bit bidirectional DQ per channel

Column command/address 8 8-bit wide column address bits

Row command/address 6 6-bit wide row address bits

DBI 16 1 DBI per 8 DQs

DM_CB 16 1 DM per 8 DQs. You can use these

PAR 4 1 parity bit per 32 DQs

DERR 4 1 data error bit per 32 DQs

Strobes 16 Separate strobes for read and write

Clock 2 Clocks address and command signals

CKE 1 Clock enable

AERR 1 Address error

pins for DM or ECC, but not both.

strobes. One differential pair per 32 DQs for read and write.

The following table lists the HBM2 signals that are common to all Pseudo Channels in each HBM2 interface. The HBM2 controller interfaces with the following signals; these signals are not available at the AXI4 user interface.

Table 2. Summary of Global HBM2 Signals

Signal Name Signal Width Notes

Reset 1 Reset input

TEMP 3 Temperature output from HBM2.

Cattrip 1 Catastrophic temperature sensor.

The Intel Stratix 10 MX HBM2 IP supports only the Pseudo Channel mode of the HBM2 specification. Pseudo Channel mode includes the following features:

• Pseudo Channel mode divides a single HBM2 channel into two individual subchannels of 64 bit I/O.

• Both Pseudo Channels share the channel’s row and column command bus, CK, and CKE inputs, but decode and execute commands individually.

• Pseudo Channel mode requires a burst length of 4.

• Address BA4 directs commands to either Pseudo Channel 0 (BA4 = 0) or Pseudo Channel 1 (BA4 = 1). The HBM2 controller handles the addressing requirements of the Pseudo Channels.

• Power-down and self-refresh are common to both Pseudo Channels, due to a shared CKE pin. Both Pseudo Channels also share the channel’s mode registers.

Each Intel Stratix 10 MX HBM2 interface supports a maximum of eight HBM2 channels. Each HBM2 channel has two AXI4 interfaces, one per Pseudo Channel. The following figure shows the flow of data from user logic to the HBM2 DRAM through the UIBSS, while selecting HBM2 channels 0 and 7.

Intel® Stratix® 10 MX HBM2 IP User Guide 8

2 Intel Stratix 10 MX HBM2 Architecture

UG-20031 | December 2017

Figure 4. Intel Stratix 10 MX HBM2 Interface Using HBM2 Channels 0 and 7 through

the UIBSS

There is one AXI interface per Pseudo Channel. The AXI4 protocol can handle concurrent writes and reads to the HBM2 controller. There is also a sideband user port per user channel pair, compliant to the Advanced Peripheral Bus (APB). The sideband provides access to user-controlled features such as refresh requests.

2.3 Intel Stratix 10 MX HBM2 Controller Architecture

The hardened HBM2 controller provides a controller per Pseudo Channel.

Each controller consists of a write and read data path and the control logic that helps to translate user commands to the HBM2 memory. The control logic accounts for the HBM2 memory specification timing and schedules commands in an efficient manner. The following figure shows a block diagram of the HBM2 controller, corresponding to channel 0. Each channel consists of two AXI ports (one per Pseudo Channel), and a sideband APB interface, which lets you issue requests, such as auto-refresh, to the HBM2.

Intel® Stratix® 10 MX HBM2 IP User Guide

2 Intel Stratix 10 MX HBM2 Architecture

Figure 5. Intel Stratix 10 MX HBM2 Controller Block Diagram

UG-20031 | December 2017

2.3.1 Intel Stratix 10 MX HBM2 Controller Details

This topic explains some of the high level HBM2 controller features.

HBM2 burst transactions

The HBM2 controller supports only the Pseudo Channel mode of accessing the HBM2 device; consequently, it can only support BL4 transactions to the DRAM. For improving efficiency, it supports the pseudo-BL8 mode, which helps to provide two back-to-back BL4 data using a given start address, similar to a BL8 transaction.

Each BL4 transaction corresponds to 4*64 bits or 32 bytes and a BL8 transaction corresponds to 64 bytes per Pseudo Channel. You can select the burst transaction mode (32 B vs 64B) through the parameter editor.

The user logic can interface to a maximum of 16 Pseudo Channels (16 AXI ports) per HBM2 interface. Each AXI port has a separate write and read interface, and can handle write and read requests concurrently at the same clock. Each write and read data interface per AXI port is 128 bits wide.

User interface vs HBM2 Interface Frequency

The user interface runs at a frequency lower than the HBM2 interface; the maximum interface frequency depends on the chosen device speed grade and the FPGA core logic frequency. The rate-matching FIFOs within the UIB subsystem handle the data transfer between the two clock domains.

Intel® Stratix® 10 MX HBM2 IP User Guide 10

2 Intel Stratix 10 MX HBM2 Architecture

UG-20031 | December 2017

The user interface runs at full rate – that is, data provided on the AXI write or read data bus on each user clock cycle corresponds to that required in one HBM2 memory clock cycle.

Command Priority

You can set command priority for a write or read command request through the AXI interface, through the qos signal in the AXI write address channel, or in the AXI read address channel. The HBM2 controller supports normal and high priority levels. The system executes commands with the same priority level in a round-robin scheme.

Starvation limit

The controller tracks how long each command waits and leaves no command unserviced in the command queue for a long period of time. The controller ensures that it serves every command efficiently.

Command scheduling

The HBM2 controller schedules the incoming commands to achieve maximum efficiency at the HBM2 interface. The HBM2 controller also follows the AXI ordering model of the AXI4 protocol specification.

Data re-ordering

The controller can reorder read data to match the order of the read requests.

Address ordering

The HBM2 controller supports different address ordering schemes that you can select for best efficiency given your use case. The chosen addressing scheme determines the order of address configurations in the AXI write and read address buses, including row address, column address, bank address, and stack ID (applicable only to the 8H devices). The HBM2 controller remaps the logical address of the command to physical memory address.

Intel® Stratix® 10 MX HBM2 IP User Guide

2 Intel Stratix 10 MX HBM2 Architecture

UG-20031 | December 2017

Thermal Control

The HBM2 controller uses the TEMP and CATTRIP outputs from the HBM2 device to manage temperature variations in the HBM2 interface.

• Temperature compensated refresh (TEMP): The HBM2 DRAM provides temperature compensated refresh information to the controller through the TEMP[2:0] pins, which defines the proper refresh rate that the DRAM expects to maintain data integrity. Absolute temperature values for each encoding are vendor-specific. The encoding on the TEMP[2:0] pins reflects the required refresh rate for the hottest device in the stack. The TEMP data updates when the temperature exceeds vendor-specified threshold levels appropriate for each refresh rate.

• Catastrophic temperature sensor (CATTRIP): The CATTRIP sensor detects whether the junction temperature of any die in the stack exceeds the catastrophic trip threshold value CATTEMP. The device vendor programs the CATTEMP to a value less than the temperature at which permanent damage to the HBM stack would occur.

If a junction temperature anywhere in the stack exceeds the CATTEMP value, the HBM stack drives the external CATTRIP pin to 1, indicating that catastrophic damage may occur. When the CATTRIP pin is at 1, the controller stops all traffic to HBM and stalls indefinitely. To resolve the overheating situation and return the CATTRIP value to 0, remove power from the device and allow sufficient time for the device to cool before again applying power.

• Thermal throttling: Thermal throttling is a controller safety feature that helps control thermal runaway if the HBM2 die overheats, preventing a catastrophic failure. You can specify the HBM2 device junction temperature at which the controller begins to throttle input commands, and the throttle ratio that determines the throttle frequency. The controller deasserts the AXI ready signals (awready, wready and arready) when it is actively throttling the input commands and data.

Refresh requests

The HBM2 controller handles HBM2 memory refresh requirements and issues refresh requests at the optimal time. The controller automatically controls refresh rates based on the temperature setting of the memory through the TEMP vector that the memory provides. You can select the HBM2 controller refresh policy, based on the frequency of refresh requests. You can choose to issue refresh commands directly, through the sideband APB interface.

Precharge policy

The HBM2 controller issues precharge commands to the HBM2 memory based on the write/read transaction address. In addition, you can issue an auto-precharge command together with a write and read command, through the AXI write address port and AXI read address port.

There are two auto-precharge modes:

• HINT – You can issue the auto-precharge request. The controller then decides when to issue the precharge command.

• FORCED – You provide auto-precharge requests through the AXI interface and the precharge request executes.

Intel® Stratix® 10 MX HBM2 IP User Guide 12

2 Intel Stratix 10 MX HBM2 Architecture

UG-20031 | December 2017

Power down enable

To conserve power, the HBM2 controller can enter power-down mode when the bus is idle for a long time. You can select this option if required.

HBM2 Controller features enabled by default

The HBM2 controller enables the following features by default:

• DBI – The DBI option supports both write and read DBI, and optimizes SI/power consumption by restricting signal switching on the HBM2 DQ bus.

• Parity – Supports command/address parity and DQ parity.

Intel Stratix 10 MX HBM2 IP User Manual

Specifications and Main Features

Frequently Asked Questions

User Manual

Contents

1 Introduction to High Bandwidth Memory

1.1 HBM2 in Intel Stratix 10 MX Devices

1.2 HBM2 DRAM Structure

1.3 Intel Stratix 10 MX HBM2 Features

1.4 Intel Stratix 10 MX HBM2 Controller Features

2 Intel Stratix 10 MX HBM2 Architecture

2.1 Intel Stratix 10 MX HBM2 Introduction

2.2 Intel Stratix 10 MX HBM2 Architecture

2.3 Intel Stratix 10 MX HBM2 Controller Architecture

2.3.1 Intel Stratix 10 MX HBM2 Controller Details