Achronix Speedster22i User Manual Interlaken

Speedster22i

Interlaken User Guide

UG032 – May 15, 2014

UG032, May 15, 2014

1

Copyright Info

Copyright © 2014 Achronix Semiconductor Corporation. All rights reserved.
Achronix is a trademark and Speedster is a registered trademark of Achronix
Semiconductor Corporation. All other trademarks are the property of their
prospective owners. All specifications subject to change without notice.

NOTICE of DISCLAIMER: The information given in this document is
believed to be accurate and reliable. However, Achronix Semiconductor
Corporation does not give any representations or warranties as to the
completeness or accuracy of such information and shall have no liability for
the use of the information contained herein. Achronix Semiconductor
Corporation reserves the right to make changes to this document and the
information contained herein at any time and without notice. All Achronix
trademarks, registered trademarks, and disclaimers are listed at

http://www.achronix.com and use of this document and the Information

contained therein is subject to such terms.

UG032, May 15, 2014

2

Table of Contents

Copyright Info .................................................................................. 2

Table of Contents ............................................................................ 3

Introduction ..................................................................................... 6

Design Overview ............................................................................. 8

Hierarchy ........................................................................................................... 9

Typical Operation ............................................................................................ 10

Flow Control .......................................................................................................... 10

Start-Up Procedure ................................................................................................ 10

Clocking .......................................................................................................... 12

TX LBUS Interface .......................................................................................... 25

Data Formatting ..................................................................................................... 26

Packet Mode ......................................................................................................... 26

Segment Mode ...................................................................................................... 26

Use of tx_bctlin ...................................................................................................... 27

RX LBUS Interface .......................................................................................... 28

Status/Control Interface ................................................................................... 30

RX Meta Frame Status .......................................................................................... 30

RX Error Status ..................................................................................................... 31

CRC32 Diagnostics Checking................................................................................ 33

Interlaken Status Messaging for the Receiver ........................................................ 33

Interlaken Status Messaging for the Transmitter .................................................... 34

Transmitter Multiple-Use Bits ................................................................................. 34

Receiver Multiple-Use Bits ..................................................................................... 34

Transmitter Flow-Control Inputs ............................................................................. 35

Register Interface ............................................................................................ 36

Description of Feature s ................................................................ 45

Lane Decommission ........................................................................................ 45

Disabling Consecutive Lanes................................................................................. 45

Consecutive Transmit Lanes ............................................................................................ 45

Consecutive Receive Lanes ............................................................................................. 45

Disabling a Single Lane ......................................................................................... 46

Single Transmit Lane ........................................................................................................ 46

Single Receive Lane ......................................................................................................... 46

Link Level Flow Control ................................................................................... 47

UG032, May 15, 2014

3

TX Rate Limiting .............................................................................................. 48

Example: Programming the Rate Limiter ............................................................... 49

Error Handling ................................................................................................. 50

Revision History .............................................................................................. 51

UG032, May 15, 2014

4

Table of Figures

Figure 1: Shows a block diagram of IIPC with SerDes and user logic interface ............... 8

Figure 2: IIPC Hierarchy .................................................................................................... 9

Figure 3: RX Clock Domains ............................................................................................ 13

Figure 4: TX Clock Domains............................................................................................. 14

Figure 5: Sample TX Waveform with a 512-bit Data Bus ................................................. 25

Figure 6: Sample RX Waveform with a 512-bit Data Bus ................................................ 28

UG032, May 15, 2014

5

Introduction

Interlaken is a scalable chip-to-chip interconnect protocol designed to enable transmission speeds
from 10Gbps to 100Gbps and beyond. Using the latest SerDes technology and a flexible protocol
layer, Interlaken minimizes the pin and power overhead of chip-to-chip interconnect and
provides a scalable solution that can be used throughout an entire system. In addition, Interlaken
uses two levels of CRC checking and a self-synchronizing data scrambler to ensure data integrity
and link robustness.

The Achronix Interlaken IP Core (IIPC) is available as hard IP in Speedster22i devices. The IIPC is
a high-performance, low-power and flexible implementation of the Interlaken Protocol. The IIPC
is compliant with the Interlaken Protocol Definition, Revision 1.2, and offers system designers
with a risk-free and quick path for adopting Interlaken as their chip-to-chip interconnect
protocol. The IIPC can be configured to use any number of serial lanes (between 4 to 12) for an
aggregate bandwidth of up to 123 Gbps.

The rest of this document describes the IIPC in detail and provides the information required for
the user to integrate the IIPC into their designs. The document assumes the reader is familiar
with the Interlaken protocol and FPGA design and methodology.

Interlaken is a very flexible and customizable protocol. IIPC supports the following features:

• Designed to take full advantage low-power design flows and methodology

• Automatic scaling of clocks to minimize power consumption

• Support for up to 10.3125Gbps SerDes data rate

• Support for 256 different logical channels

• Robust error condition detection and recovery

• Flexible SerDes interface to accommodate different I/O implementations

• Data striping and de-striping across 4 to 12 lanes

• Programmable BurstMax, BurstShort and MetaFrameSize parameters

• 64/67 encoding and decoding

• Automatic word and lane alignment

• Self-synchronizing data scrambler

• Data bus width of 512 bits

• CRC24 generation and checking for burst data integrity

• CRC32 generation and checking for lane data integrity

• Data scrambling and disparity tracking to minimize baseline wander and maintain DC

balance

• Support for all Synchronization, Scrambler State, Diagnostic, and Skip Word Block Types

• Programmable Rate Limiting circuitry

• Segment-mode and Packet-mode transmission format

UG032, May 15, 2014

6

• Segment-mode and Packet-mode receive format

• BurstMax size can be programmed from 64 bytes to 256 bytes in steps of 64 bytes

• Support for minimum BurstShort requirement of 64 bytes up until 256 bytes in steps of

64 bytes but not exceeding BurstMax

• In-band flow control

• Support for link-level flow control

• Flow control mechanism supports stopping packets in mid-stream – head of line

blocking

• Rate matching with granularity of 1 Gbps

• Meta Frame Length programmable between 128 to 8K words

• Support for status messaging

• Lane decommissioning and resiliency

UG032, May 15, 2014

7

Design Overview

Figure 1 shows a block diagram of the IIPC with SerDes and user logic interfaces implemented in
Speedster22i. The right hand side is the user interface and the left hand side is the SerDes
interface. The IIPC is shown in green and the SerDes blocks in orange.

Figure 1: Shows a block diagram of IIPC with SerDes and user logic interface

UG032, May 15, 2014

8

Hierarchy

Interlaken_1_serdes

ACX_INTERLAKEN_TOP

(IIPC core)

Interlaken_1

The interfaced IIPC and SerDes IP is configured using the Achronix Cad Environment (ACE)
Interlaken GUI.

The upper-levels of hierarchy are shown in Figure 2. Note that the text in Figure 2 represents the
name of the module at that level of hierarchy.

Figure 2: IIPC Hierarchy

The top-level wrapper module, Interlaken_1, contains the instantiated hard IIPC block and an
instantiated hard SerDes block for each lane. The module Interlaken_1_serdes wraps the Achronix
SerDes block and sets configuration parameters needed for proper Interlaken operation.

User’s Tasks:

• The ACE Interlaken GUI should be used to configure the Interlaken core. This will

automatically instantiate the SerDes lanes and produces a single top level wrapper that
can be included in your design.

• Connect the Local Bus (LBUS) user-side interface at the fabric to the wrapper

• Provide the appropriate clock signals to the wrapper

UG032, May 15, 2014

9

Typical Operation

The IIPC can be used in a variety of applications, and provides the user all of the flexibility
offered by the Interlaken Protocol. The IIPC handles all protocol-related functions needed to
communicate to another device’s Interlaken interface. All handshaking, synchronizing and error
checking are handled by the IIPC. The LBUS is designed to match commonly used packet bus

protocols made common by the SPI4.2 protocol; a detailed description is given in the User

Interfaces section.

User’s Task:

• Provide packet data via the Local Bus (LBUS) TX interface, and receive packet data from

the LBUS RX interface.

Flow Control

Flow control information is automatically extracted by the RX path of the IIPC and presented to
the user. Also, the IIPC’s TX path consists of a single pipeline with a single memory buffer.

User’s Tasks:

1. Build the scheduling mechanism external to the IIPC to mux data from different logical

channels.

2. Monitor the flow control information and ensure proper data transmission through the

IIPC

Start-Up Procedure

The following lists the different operation steps upon IIPC power-up:

1. After the device is powered up and the reset procedure completed, the IIPC TX path

starts transmitting Control/Idle words in order to align and synchronize the receiving
device’s Interlaken interface. Similarly, the IIPC RX path listens for Control/Idle words
and goes through its own synchronization procedure.

2. User’s Task: Set all of the flow control input to the IIPC TX path to the XOFF state to

prevent any real data transfer.

3. The RX path will eventually get aligned and synchronized and signal the user logic that

synchronization is complete.

4. User’s Task: Turn the flow control information from XOFF to XON for any of the

channels that are ready to accept data.

5. Similarly when the other device is ready to receive data, it will send XON information to

the IIPC which in turn will tell the user logic which channels can be used to transmit data
on.

The above list describes a simple and easily-implementable procedure by which to initially
configure the IIPC. The user needs to build a scheduler only to mux data amongst the different
logical channels, and use the flow control information output by the IIPC to manage the

UG032, May 15, 2014

10

scheduling function. The user need not be concerned about any of the lower level Interlaken
Protocol details.

UG032, May 15, 2014

11

Clocking

The IIPC has three major clock domains:

1. LBUS clock Domain

• The clk input port is used to clock the protocol processing of the IIPC. This

includes all logic in the TX and RX paths responsible for protocol layer
processing including Control Word, Meta frame and the LBUS interface.

• The frequency of the clk domain is 470MHz

2. RX SerDes clock Domain

• Each SerDes is assumed to provide its recovered clock to the IIPC. These clocks

are connected to the rx_serdes_clk[11:0] input pins and are used to clock the per­lane logic of each lane. IIPC synchronizes the received data from all of the SerDes
to the LBUS clock domain.

• The frequency of these domains is calculated by simply dividing the serial bit

rate by the parallel bus width (20-bit) of the SerDes block. For example, if the
serial bit rate is 6.25Gbps, the rx_serdes_clk[11:0] will have a frequency of
(6250Mbps/20) = 312.5MHz.

3. TX SerDes Reference Clock Domain

• The TX SerDes domain consists of logic that is operated on the clock domain

associated with each TX SerDes. All of the SerDes must be clocked using the
same reference clock source to ensure frequency matching between the lanes. To
take care of phase differences between lanes, IIPC generates the transmit data for
all SerDes interfaces using one common clock called tx_serdes_refclk.

• The frequency of this domain is calculated by simply dividing the serial bit rate

by the parallel bus width (20-bit) of the SerDes block. For example, if the serial
bit rate is 6.25Gbps, the tx_serdes_refclk will have a frequency of (6250Mbps/20)
= 312.5MHz.

Figure 3 shows the different clock domains in the RX direction along with their associated clock
inputs. Figure 4 shows the different clock domains in the TX direction along with their associated
clock inputs. The IIPC handles all clock domain crossings.

UG032, May 15, 2014

12

hs

_if

per_lane_

rx

rx

_destriping

clk

serDes

IIPC Core

hs_

if

per

_

lane

_

rx

serDes

hs

_

if

per_lane_rx

serDes

hs_if

per

_lane

_

rx

serDes

hs

_

if

per_

lane_rx

serDes

hs_if

per

_lane_

rx

serDes

hs_

if

per_lane

_rx

serDes

hs_if

per

_lane_

rx

serDes

hs_

if

per

_lane

_

rx

serDes

hs_

if

per

_lane_rx

serDes

hs_if

per_

lane_rx

serDes

hs_

if

per

_lane_rx

serDes

Interface to user logic in the FPGA Core

rx_serdes_clk[0]

rx_

serdes

_clk

[

1

]

rx_serdes_clk[2]

rx

_

serdes_

clk[

3

]

rx

_serdes_

clk[

4]

rx

_serdes_

clk[

5]

rx

_serdes_

clk[6

]

rx

_serdes_

clk[7]

rx_

serdes

_clk[

8]

rx

_serdes

_clk

[9]

rx

_serdes

_

clk[10]

rx

_

serdes_

clk[11

]

Figure 3: RX Clock Domains

UG032, May 15, 2014

13

hs_if per_lane

_tx

tx_striping

clktx_serdes_refclk

serDes

IIPC Core

hs_if

per_lane_txserDes

hs_if

per_lane_tx

serDes

hs_if per_lane_tx

serDes

hs_if per_lane_txserDes

hs_if per_lane_tx

serDes

hs_if

per_lane_tx

serDes

hs_if

per_lane_tx

serDes

hs_if

per_lane_txserDes

hs_if per_lane_tx

serDes

hs_if per_lane_txserDes

hs_if

per_lane_tx

serDes

Interface to user logic in the FPGA Core

Figure 4: TX Clock Domains

UG032, May 15, 2014

14

Port List

synchronized to the positive edge of the

Table 1 shows the port list for the IIPC. More detail on the use of each signal is given in the User

Interfaces section.

Table 1 – Port Description

Name Direction Clock Description

Register Interface

sbus_req

sbus_datain[1:0]

sbus_ack

sbus_dataout[1:0]

rx_serdes_data[19:0
]

Input sbus_clk

Input sbus_clk

Output sbus_clk

Output sbus_clk

Transceiver I/O

Input rx_serdes_clk

Asserted for 9-cycles in case of read and for
11-cycles in case of write

Carries read/write indication, address and
data to write

Acknowledgment from register i/f once
read or write is completed thru SBUS.
During write it is valid for one cycle to
indicate the end of the transfer. This is
asserted for 4-cycles to validate 8-bit data at
the end of read.

Contains read data for 4-cycles when
o_sbus_ack is asserted.

Data bus from the SerDes. There are 12
rx_serdes_data buses; one bus for each
SerDes lane and each bus has 20 bits.

By definition, bit [19] is the first bit received
by the IIPC. Bit [0] is the last bit received.

tx_serdes_data[19:0]

rx_serdes_clk[11:0] Input

UG032, May 15, 2014

Output

tx_serdes_ref
clk

Data bus to the SerDes . There are 12
tx_serdes_data buses; one bus for each
SerDes lane and each bus has 20 bits.

By definition, bit [19] is the first bit
transmitted by the IIPC. Bit [0] is the last bit
transmitted.

Recovered clock of each SerDes lane. The
rx_serdes_data bus for each lane

15

Name Direction Clock Description

corresponding bit of this bus.

rx_serdes_resetn[11:
0]

Input

Reset for each RX SerDes lane. The
recovered clock for each SerDes lane has
associated with it an active-low reset. This
signal is low until transmit and receive
data-paths of each SerDes are ready.

tx_serdes_refclk Input

tx_serdes_refclk_res
etn

Input

LBUS Interface – Clock/Reset Signals

clk Input

rx_resetn Input

Common clock for the all Tx SerDes
interfaces.

Active low reset for TX Reference clock.
This signal is low until the PLL in the
SerDes is locked and the transmit data-path
is ready in lane #0.

Local bus clock. All signals between the
IIPC and the user side logic are
synchronized to the positive edge of this
signal.

Asynchronous reset for the RX circuits.
This signal is active low (0 = reset). Initially
this reset is applied from the board.
Subsequently it can be applied by the user
dynamically or based on the readiness of
SerDes receive and transmit paths. The
IIPC handles synchronizing the rx_resetn
input to the appropriate clock domains
within the IIPC.

tx_resetn Input

rx_dataout[511:0] Output clk

rx_chanout[7:0] Output clk

UG032, May 15, 2014

Asynchronous reset for the TX circuits. This
signal is active low (0 = reset). Initially this
reset is applied from the board.
Subsequently it can be applied by the user
dynamically or based on the readiness of
SerDes PLL and transmit path. The IIPC
handles synchronizing the tx_resetn input
to the appropriate clock domains within the
IIPC.

LBUS Interface – RX Path Signals

Receive LBUS Data. The value of the bus is
only valid in cycles that rx_enaout is
sampled as 1.

Receive Channel Number. The bus

16