Altera Stratix V Avalon-ST User Manual

Stratix V Avalon-ST Interface with SR-IOV PCIe Solutions

User Guide

Last updated for Altera Complete Design Suite: 14.1

Subscribe Send Feedback

UG-01097_sriov

2014.12.15

101 Innovation Drive San Jose, CA 95134

www.altera.com

2014.12.15

Application

Layer

(User Logic)

Avalon-ST

Interface

PCIe Hard IP with SR-IOV

Block

PIPE

Interface

PHY IP Core

for PCIe

(PCS/PMA)

Serial Data

Transmission

www.altera.com

101 Innovation Drive, San Jose, CA 95134

Datasheet

UG-01097_sriov

Send Feedback

Stratix V Avalon-ST Interface with SR-IOV for PCIe Datasheet

Altera® Stratix® V FPGAs include a configurable, hardened protocol stack for PCI Express compliant with PCI Express Base Specification 2.1 or 3.0. The Stratix V Hard IP for PCI Express with Single Root I/O Virtualization (SR-IOV) IP core consists of this hardened protocol stack and the SR-IOV soft logic. The SR-IOV soft logic uses Configuration Space Bypass mode to bypass the hardened Configu‐ ration Space. It implements the following functions in soft logic:

• Configuration Spaces for up to 2 PCIe Physical Functions (PFs) and a maximum of 128 Virtual Functions (VFs) for both PFs

• BAR checking logic

• Support for the following interrupt types:

• MSI for PFs

• MSI-X for PFs and VFs

• Legacy interrupts for PFs

• Support for Advanced Error Reporting (AER) for PFs

• Support for Function Level Reset (FLR) for PFs and VFs

• Support for x2, x4, and x8 links using a 128- or 256-bit Avalon-ST datapath

that is

For details of the Configuration Space Bypass mode interface refer to the Configuration Space Bypass

Mode Interface Signals in the Stratix V Hard IP for PCI Express User Guide for the Avalon Streaming Interface

Figure 1-1: Stratix V PCIe Variant with SR-IOV

The following figure shows the high-level modules and connecting interfaces for this variant.

2014 Altera Corporation. All rights reserved. ALTERA, ARRIA, CYCLONE, ENPIRION, MAX, MEGACORE, NIOS, QUARTUS and STRATIX words and logos are trademarks of Altera Corporation and registered in the U.S. Patent and Trademark Office and in other countries. All other words and logos identified as trademarks or service marks are the property of their respective holders as described at www.altera.com/common/legal.html. Altera warrants performance of its semiconductor products to current specifications in accordance with Altera's standard warranty, but reserves the right to make changes to any products and services at any time without notice. Altera assumes no responsibility or liability arising out of the application or use of any information, product, or service described herein except as expressly agreed to in writing by Altera. Altera customers are advised to obtain the latest version of device specifications before relying on any published information and before placing orders for products or services.

ISO 9001:2008 Registered

1-2

Features

UG-01097_sriov

2014.12.15

Table 1-1: PCI Express Data Throughput

The following table shows the aggregate bandwidth of a PCI Express link for Gen1, Gen2, and Gen3 for supported link widths. The protocol specifies 2.5 giga-transfers per second for Gen1, 5.0 giga-transfers per second for Gen2, and 8.0 giga-transfers per second for Gen3. This table provides bandwidths for a single transmit (TX) or receive (RX) channel. The numbers double for duplex operation. Gen1 and Gen2 use 8B/10B encoding which introduces a 20% overhead. In contrast, Gen3 uses 128b/130b encoding which reduces the data throughput lost to encoding to less than 1%.

Link Width

×2 ×4 ×8

PCI Express Gen1 (2.5 Gbps) - 128-bit interface

PCI Express Gen2 (5.0 Gbps) - 128-bit interface

PCI Express Gen2 (5.0 Gbps) - 256-bit interface

N/A N/A

N/A

N/A N/A

PCI Express Gen3 (8.0 Gbps) - 128-bit interface 15.75 31.51

PCI Express Gen3 (8.0 Gbps) - 256-bit interface

Related Information

N/A N/A

• PCI Express Base Specification 2.1 or 3.0

• Single Root I/O Virtualization and Sharing Specification Revision 1.1.

• Stratix V Avalon-ST Interface for PCIe Solutions User Guide

• Creating a System with Qsys

Features

New features in the Quartus® II 14.1 software release:

• Reduced Quartus II compilation warnings by 50%.

16 32

N/A

The Stratix V Hard IP for PCI Express with SR-IOV supports the following features:

• Complete protocol stack including the Transaction, Data Link, and Physical Layers implemented as

• Support for ×2, ×4, and ×8 configurations with Gen1, Gen2, or Gen3 lane rates for Endpoints.

• Dedicated 16 KByte receive buffer.

• Optional hard reset controller for Gen2.

• Qsys example designs demonstrating parameterization, design modules, and connectivity.

• Extended credit allocation settings to better optimize the RX buffer space based on application type.

• End-to-end cyclic redundancy code (ECRC) generation and checking and advanced error reporting

Altera Corporation

hard IP.

Downtrains to appropriate configuration when plugged into a lower bandwidth configuration, including Gen1 x1, Gen1 x2, and so on.

(AER) for high reliability applications.

Datasheet

Send Feedback

UG-01097_sriov

2014.12.15

• Support for Configuration Space Bypass Mode, allowing you to design a custom Configuration Space and support multiple functions.

• Support for Gen3 PIPE simulation.

• Easy to use:

• Flexible configuration.

• No license requirement.

• Example designs to get started.

Table 1-2: Feature Comparison for all Hard IP for PCI Express IP Cores

The table compares the features of the four Hard IP for PCI Express IP Cores.

Features

1-3

Feature Avalon‑ST Interface Avalon‑MM

Interface

Avalon‑MM DMA Avalon‑ST Interface with SR-

IP Core License Free Free Free Free

Native

Supported Supported Supported Supported

Endpoint

Legacy Endpoint

(1)

Supported Not Supported Not Supported Not Supported

Root port Supported Supported Not Supported Not Supported

Gen1 ×1, ×2, ×4, ×8 ×1, ×2, ×4, ×8 Not Supported

Gen2 ×1, ×2, ×4, ×8 ×1, ×2, ×4, ×8 ×4, ×8

Gen3 ×1, ×2, ×4, ×8 ×1, ×2, ×4 ×4, ×8

64-bit Applica‐

Supported Supported Not supported Not supported

×8

×4, ×8

×2, ×4, ×8

tion Layer interface

IOV

(1)

Datasheet

128-bit

Supported Supported Supported Supported Application Layer interface

256-bit

Supported Not Supported Supported Supported Application Layer interface

Not recommended for new designs.

Send Feedback

Altera Corporation

1-4

Features

UG-01097_sriov

2014.12.15

Feature Avalon‑ST Interface Avalon‑MM

Interface

Transaction Layer Packet type (TLP)

• Memory Read Request

• Memory Read RequestLocked

• Memory Write Request

• I/O Read Request

• I/O Write Request

• Configuration Read Request (Root Port)

• Configuration Write Request (Root Port)

• Message Request

• Message Request with Data Payload

• Completion Message

• Completion with Data

• Memory Read Request

• Memory Write Request

• I/O Read Request—Root Port only

• I/O Write Request—Root Port only

• Configuration Read Request (Root Port)

• Configuration Write Request (Root Port)

• Completion Message

• Completion with Data

• Memory Read Request (single dword)

• Memory Write Request (single dword)

• Completion for Locked Read without Data

Avalon‑MM DMA Avalon‑ST Interface with SR-

IOV

• Memory Read Request

• Memory Write Request

• Completion Message

• Completion with Data

• Memory Read Request

• Memory Write Request

• Configuration Read Request (from Root Port)

• Configuration Write Request (from Root Port)

• Message Request

• Completion Message

• Completion with Data

Payload size

Number of tags supported for non-posted requests

62.5 MHz clock Supported Supported Not Supported Not Supported

Altera Corporation

128–2048 bytes 128–256 bytes 128, 256, 512 bytes 128–256 bytes

256 8 16 256

Datasheet

Send Feedback

UG-01097_sriov

2014.12.15

Features

1-5

Feature Avalon‑ST Interface Avalon‑MM

Interface

Out-of-order

Not supported Supported Supported Not supported completions (transparent to the Application Layer)

Requests that

Not supported Supported Supported Supported cross 4 KByte address boundary (transparent to the Application Layer)

Polarity

Supported Supported Supported Supported Inversion of PIPE interface signals

Avalon‑MM DMA Avalon‑ST Interface with SR-

IOV

ECRC

Supported Not supported Not supported Not supported forwarding on RX and TX

Number of MSI requests

1, 2, 4, 8, 16, or 32 1, 2, 4, 8, 16, or 32 1, 2, 4, 8, 16, or 32 1, 2, 4, 8, 16, or 32 (for

Physical Functions)

MSI-X Supported Supported Supported Supported

Legacy

Supported Supported Supported Supported interrupts

Expansion

Supported Not supported Not supported Not supported ROM

The Stratix V Avalon-ST Interface with SR-IOV PCIe Solutions User Guide explains how to use this IP core and not the PCI Express protocol. Although there is inevitable overlap between these two purposes, use this document only in conjunction with an understanding of the PCI Express Base Specification.

Note:

This release provides separate user guides for the different variants. The Related Information provides links to all versions.

Related Information

Datasheet

• Stratix V Avalon-MM Interface for PCIe Solutions User Guide

• Stratix V Avalon-ST Interface for PCIe Solutions User Guide

• Stratix V Avalon-ST Interface with SR-IOV for PCIe Solutions User Guide

Altera Corporation

Send Feedback

1-6

Release Information

• V-Series Avalon-MM DMA Interface for PCIe Solutions User Guide

Release Information

Table 1-3: Hard IP for PCI Express Release Information

Item Description

Version 14.1

Release Date December 2014

Ordering Codes No ordering code is required

UG-01097_sriov

2014.12.15

Product IDs

Vendor ID

The Product ID and Vendor ID are not required because this IP core does not require a license.

Device Family Support

Table 1-4: Device Family Support

Device Family Support

Stratix V Preliminary. The IP core is verified with prelimi‐

nary timing models for this device family. The IP core meets all functional requirements, but might still be undergoing timing analysis for the device family. It can be used in production designs with caution.

Other device families Refer to the Related Information below for other

device families:

Related Information

• Arria V Avalon-MM Interface for PCIe Solutions User Guide

• Arria V Avalon-ST Interface for PCIe Solutions User Guide

• Arria V GZ Avalon-MM Interface for PCIe Solutions User Guide

• Arria V GZ Avalon-ST Interface for PCIe Solutions User Guide

• Arria 10 Avalon-MM Interface for PCIe Solutions User Guide

• Arria 10 Avalon-MM DMA Interface for PCIe Solutions User Guide

• Arria 10 Avalon-ST Interface for PCIe Solutions User Guide

• Cyclone V Avalon-MM Interface for PCIe Solutions User Guide

• Cyclone V Avalon-ST Interface for PCIe Solutions User Guide

• IP Compiler for PCI Express User Guide

Altera Corporation

Datasheet

Send Feedback

UG-01097_sriov

2014.12.15

Example Designs

Altera provides example designs to familiarize you with the available functionality. Each design connects the device under test (DUT) to an application programming platform (APP), labeled APPs in the figure below. Certain critical parameters of the APPs component are set to match the values of the DUT. If you change these parameters, you must change the APPs component to match. You can change the values for all other parameters of the DUT without editing the APPs component.

In this example design, the following parameters must be set to match the values set in the DUT:

• Targeted Device Family

• Lanes

• Lane Rate

• Application Clock Rate

• Port type

• Application Interface

• Tags supported

• Maximum payload size

• Total PFs

• Total VFs

Example Designs

1-7

The following Qsys example designs are available for the Stratix V Hard IP for PCI Express with SR-IOV. You can download them from the <install_dir>/ ip/altera/altera_pcie/altera_pcie_sriov/example_design/ directory:

• sriov_top_dma_gen2_x8_128b.qsys

• sriov_top_dma_gen2_x8_256b.qsys

• sriov_top_dma_gen3_x8_256b.qsys

• sriov_top_target_gen2_x8_256b_2pf.qsys

• sriov_top_target_gen3_x8_256b_1pf_32vf.qsys

• sriov_top_target_gen3_x8_256b_2pf_128vf.qsys

• sriov_top_target_gen3_x8_256b_2pf_4vf.qsys

• sriov_top_target_gen3_x8_256b_1pf_4vf_avmm.qsys

Related Information

Getting Started with the SR-IOV DMA Example Design on page 2-1

Debug Features

Debug features allow observation and control of the Hard IP for faster debugging of system-level problems.

Related Information

Debugging on page 12-1

Datasheet

Altera Corporation

Send Feedback

1-8

IP Core Verification

To ensure compliance with the PCI Express specification, Altera performs extensive verification. The simulation environment uses multiple testbenches that consist of industry-standard bus functional models (BFMs) driving the PCI Express link interface. Altera performs the following tests in the simulation environment:

• Directed and pseudorandom stimuli are applied to test the Application Layer interface, Configuration Space, and all types and sizes of TLPs

• Error injection tests that inject errors in the link, TLPs, and Data Link Layer Packets (DLLPs), and check for the proper responses

• PCI-SIG® Compliance Checklist tests that specifically test the items in the checklist

• Random tests that test a wide range of traffic patterns

Altera provides the following two example designs that you can leverage to test your PCBs and complete compliance base board testing (CBB testing) at PCI-SIG.

Related Information

• PCI SIG Gen3 x8 Merged Design - Stratix V

• PCI SIG Gen2 x8 Merged Design - Stratix V

UG-01097_sriov

2014.12.15

Compatibility Testing Environment

Altera has performed significant hardware testing to ensure a reliable solution. In addition, Altera internally tests every release with motherboards and PCI Express switches from a variety of manufac‐ turers. All PCI-SIG compliance tests are run with each IP core release.

Performance and Resource Utilization

Because the PCIe protocol stack is implemented in hardened logic, it uses less than 1% of device resources.

Table 1-5: Performance and Resource Utilization Stratix V Avalon-MM DMA for PCI Express

Number of PFs and VFs

2 PFs 2000 14 4800

1 PF, 4 VFs 3000 14 5450

1 PF, 32 VFs 3250 14 5950 2 PFs, 64 VFs 3650 14 6550

ALMs M20K Memory Blocks Logic Registers

2 PFs, 128 VFs 6450 14 9900

Note: Soft calibration of the transceiver module requires additional logic. The amount of logic required

Altera Corporation

depends upon the configuration.

Datasheet

Send Feedback

UG-01097_sriov

2014.12.15

Related Information

Recommended Speed Grades

Fitter Resources Reports

Recommended Speed Grades

Table 1-6: Stratix V Recommended Speed Grades for All SR-IOV Configurations

Altera recommends setting the Quartus II Analysis & Synthesis Settings Optimization Technique to Speed when the Application Layer clock frequency is 250 MHz. For information about optimizing synthesis, refer to “Setting Up and Running Analysis and Synthesis in Quartus II Help. For more information about how to effect the

Optimization Technique settings, refer to Area and Timing Optimization in volume 2 of the Quartus II Handbook. Refer to the Related Links below.

1-9

Link Rate Link Width Interface

Width

Application Clock

Frequency (MHz)

Recommended Speed Grades

Gen1 ×8 128 Bits 125 –1, –2, –3, –4

×4 128 bits 125 –1, –2, –3, –4

Gen2

×8 128 bits 250 –1, –2, –3

×8 256 bits 125 –1, –2, –3, –4

×2 128 bits 125 –1, –2, –3, –4

×4 128 bits 250 –1, –2, –3

Gen3

×4 256 bits 125 –1, –2, –3,–4

×8 256 bits 250 –1, –2, –3

Related Information

• Area and Timing Optimization

• Altera Software Installation and Licensing Manual

• Setting up and Running Analysis and Synthesis

(2)

Steps in Creating a Design for PCI Express

Before you begin

Select the PCIe variant that best meets your design requirements.

(2)

The -4 speed grade is also possible for this configuration; however, it requires significant effort by the end user to close timing.

Datasheet

Send Feedback

Altera Corporation

1-10

Steps in Creating a Design for PCI Express

UG-01097_sriov

2014.12.15

• Is your design an Endpoint or Root Port?

• What Generation do you intend to implement?

• What link width do you intend to implement?

• What bandwidth does your application require?

• Does your design require CvP?

1. Select parameters for that variant.

2. Simulate using an Altera-provided example design. All of Altera's PCI Express example designs are

available under <install_dir>/ip/altera/altera_pcie/. Alternatively, create a simulation model and use your own custom or third-party BFM. The Qsys Generate menu generates simulation models. Altera supports ModelSim®-Altera for all IP. The PCIe cores support the Aldec RivieraPro, Cadence NCsim, Mentor Graphics ModelSim, and Synopsys VCS and VCS-MX simulators.

3. Compile your design using the Quartus II software. If the versions of your design and the Quartus II software you are running do not match, regenerate your PCIe design.

4. Download your design to an Altera development board or your own PCB. Click on the All Develop‐ ment Kits link below for a list of Altera's development boards.

5. Test the hardware. You can use Altera's SignalTap® II Logic Analyzer or a third-party protocol analyzer to observe behavior.

6. Substitute your Application Layer logic for the Application Layer logic in Altera's testbench. Then repeat Steps 3–6. In Altera's testbenches, the PCIe core is typically called the DUT (device under test). The Application Layer logic is typically called APPS.

Related Information

• Parameter Settings on page 3-1

• Getting Started with the SR-IOV DMA Example Design on page 2-1

• All Development Kits

Altera Corporation

Datasheet

Send Feedback

Getting Started with the SR-IOV DMA Example

www.altera.com

101 Innovation Drive, San Jose, CA 95134

Design

2014.12.15

UG-01097_sriov

The SR-IOV example design consists of an SR-IOV bridge configured for one Physical Function (PF) and four Virtual Functions (VFs). Each VF connects to a read DMA and a write DMA engine. The examples design simulates the Transaction, Data Link, and Physical Layers using the Altera Root Port BFM. It also supports Quartus II compilation.

The SR-IOV Qsys example design includes three Qsys subsystems. The top-level Qsys system comprises the following components:

• DUT: This is the Stratix VHard IP for PCI Express with SR-IOV.

• APPs: This component is a Qsys subsystem that implements a highly efficient DMA engine. Each VF has separate descriptor controllers for read DMA and write DMA descriptors. The read DMA and write DMA routers arbitrate requests from the descriptor controllers. They forward the selected request to the read DMA and write DMA modules. The read DMA transfers large blocks of data from the Avalon-ST (SR-IOV) domain to the Avalon-MM (Qsys). The write DMA Write module transfers large blocks of data from the Avalon-MM domain to the Avalon-ST domain. Refer to the SR-IOV Example Design Block Diagram block diagram below.

In addition to high performance data transfer, the Read DMA and Write DMA modules ensure that the requests on the PCI link adhere to the PCI Express Base Specification, 3.0. The read and write DMA modules also perform the following functions:

Send Feedback

• Divide the original request into multiple requests to avoid crossing 4KByte boundaries.

• Divide the original request into multiple requests to ensure that the maximum payload size is equal

to or smaller than the maximum payload size for write.

• Divide the original request into multiple requests to ensure that the maximum read size is equal to

or smaller than the maximum read request size.

• Supports out-of-order completions when the original request is divided into multiple requests to

adhere to the maximum payload size.

• Altera PCIe Reconfig Driver IP Core: This Avalon-MM master drives the Transceiver Reconfiguration Controller.

• Transceiver Reconfiguration Controller IP Core: The Transceiver Reconfiguration Controller dynamically reconfigures analog settings to improve signal quality. For Gen1 and Gen2 data rates, the Transceiver Reconfiguration performs offset cancellation and PLL calibration. For the Gen3 data rate, the pcie_reconfig_driver_0 performs AEQ through the Transceiver Reconfiguration Controller.

ISO 9001:2008 Registered

Hard IP for PCI Express

SR-IOV Bridge

Rd_DC0 Rd_DC1 Rd_DC2 Rd_DC3

Read DMA Router

APPs - sriov_dma_app_g3x8_256b.qsys

sriov_top_dma_gen3_x8_256.qsys

rddc_ctl - rddc_ctl_256b.qsys wrdc_ctl - wrdc_ctl_256b.qsys

Wr_DC0 Wr_DC1 Wr_DC2 Wr_DC3

Write DMA Router

User Application Logic (On-Chip Memories)

DMA Write

TX Slave

RX Master DMA Read

2-2

Generating the Example Design Testbench

Figure 2-1: SR-IOV Example Design Block Diagram

UG-01097_sriov

2014.12.15

Related Information

Stratix V Hard IP for PCI Express User Guide for the Avalon Memory-Mapped Interface with DMA

Generating the Example Design Testbench

Follow these steps to generate the SR-IOV DMA example design testbench:

1. Copy <install_dir>/ ip/altera/altera_pcie/altera_pcie_sriov/example_design/sriov_top_dma_gen3_x8_256b.qsys

to your working directory. This top-level Qsys design includes three subsystems.

Altera Corporation

Qsys Subsystem Description

sriov_dma_app_g3x8_ 256b.qsys

rddc_ctl_256b.qsys This subsystem implements the Read Descriptor Controller for 4 Read

This subsystem implements of the Read DMA read and Write DMA modules and the Read and Write Descriptor Controllers. the DMA engine.

DMA channels.

Getting Started with the SR-IOV DMA Example Design

Send Feedback

UG-01097_sriov

2014.12.15

wrdc_ctl_256b.qsys This subsystem implements the Write Descriptor Controller for 4 Write

Figure 2-2: Top-Level Qsys System for SR-IOV Gen3 x8 DMA Example Design

Generating the Example Design Testbench

Qsys Subsystem Description

DMA channels.

Note: File names that include 256b have a 256-bit interface to the Application Layer. File names that

include 128b have a 128-bit interface to the Application Layer.

2. Rename the top-level Qsys file, sriov_top_dma_gen3_x8_256b.qsys, to top.qsys.

3. In your working directory, start Qsys, by typing the following command:

qsys-edit

4. Open top.qsys.

The following figure shows the Qsys system.

2-3

5. On the Generate menu, select Generate Testbench System. The Generation dialog box appears.

Getting Started with the SR-IOV DMA Example Design

Send Feedback

Altera Corporation

2-4

Generating the Example Design Testbench

6. Specify the following parameters:

Table 2-1: Parameters to Specify on the Generation Menu in Qsys

Parameter Value

Testbench System

Create testbench Qsys system Standard, BFMs for standard Avalon interfaces

Create testbench simulation model Verilog. This option generates simulation files for

the testbench.

Allow mixed-language simulation Leave this option off.

Output Directory

UG-01097_sriov

2014.12.15

Path

Testbench

working_dir/

working_dir/testbench/

7. Click Generate. Qsys generates the testbench.

8. To generate files for Quartus II compilation, on the Generate menu, select Generate HDL. The Generation dialog box appears.

9. Specify the following parameters:

Table 2-2: Parameters to Specify on the Generation Menu in Qsys

Parameter Value

Verilog

Create HDL design files for synthesis Verilog.

Create timing and resource estimates for third-

Leave this option off.

party EDA synthesis tools

Create block symbol file (.bsf) Leave this option on.

Create simulation model.

Allow mixed language simulation.

Altera Corporation

Simulation

None . (You created the simulation model when

you generated the testbench.) Leave this option off.

Output Directory

Getting Started with the SR-IOV DMA Example Design

Send Feedback

UG-01097_sriov

2014.12.15

Understanding the Generated Files and Directories

Parameter Value

2-5

Path

working_dir/top

10.Click Generate.

11.On the File menu, click Save.

Understanding the Generated Files and Directories

Table 2-3: Qsys Generation Output Files

Directory Description

<testbench_dir>/<variant_name>/testbench

<testbench_dir>/<variant_name>/testbench/<cad_ vendor>

<testbench_dir>/<variant_name>/testbench/<variant_ name>_tb/simulation/submodules

Includes testbench subdirectories for the Aldec, Cadence, Mentor, and Synopsys simulation tools with the required libraries and simulation scripts.

Includes the HDL source files and scripts for the simulation testbench.

Includes the HDL files for simulation.

Simulating the SR--IOV Example Design

Follow these steps to simulate the Qsys system using ModelSim:

1. In a terminal window, change to the <working_dir>/sim/mentor directory.

2. Start the ModelSim simulator by typing vsim.

3. To compile the simulation, type the following commands in the terminal window:

• do msim_setup.tcl (The msim_setup.tcl file defines aliases.

• ld_debug (The ld_debug command argument stops optimizations, improving visibility in the ModelSim waveforms. )

• run -all

Running A Gate-Level Simulation

The PCI Express testbenches run simulations at the register transfer level (RTL). However, it is possible to create you own gate-level simulations. Contact your Altera Sales Representative for instructions and an example that illustrate how to create a gate-level simulation from the RTL testbench.

Getting Started with the SR-IOV DMA Example Design

Send Feedback

Altera Corporation

Rd_DC0 Rd_DC1 Rd_DC2 Rd_DC3

Read DMA Router

APPs - sriov_dma_app_g3x8_256b.qsys

rddc_ctl - rddc_ctl_256b.qsys wrdc_ctl - wrdc_ctl_256b.qsys

Wr_DC0 Wr_DC1 Wr_DC2 Wr_DC3

Write DMA Router

DMA Write

TX Slave

RX Master DMA Read

Stratix V Hard IP for PCI Express

SR-IOV Bridge

User Application Logic (On-Chip Memories)

sriov_top_dma_gen3_x8_256.qsys

2-6

Understanding the DMA Functionality

The following figures illustrate the DMA functionality using numbered steps.

Figure 2-3: Steps to Fetch Descriptor Table from Host Memory

UG-01097_sriov

2014.12.15

Fetching the Descriptor Table entries includes the following steps:

1. The host sets up descriptor controller register table using the RX master interface.

2. The Descriptor Controller instructs the DMA Read module to fetch the descriptor instruction entries.

3. The Host returns descriptor instruction entries to the Descriptor Controller.

4. In response to the Descriptor Controller instruction, the DMA Read drives a Memory Read Request to

the Hard IP.

Altera Corporation

Getting Started with the SR-IOV DMA Example Design

Send Feedback

sriov_top_dma_gen3_x8_256.qsys

Hard IP for PCI Express

SR-IOV Bridge

Rd_DC0 Rd_DC1 Rd_DC2 Rd_DC3

Read DMA Router

APPs - sriov_dma_app_g3x8_256b.qsys

rddc_ctl - rddc_ctl_256b.qsys wrdc_ctl - wrdc_ctl_256b.qsys

Wr_DC0 Wr_DC1 Wr_DC2 Wr_DC3

Write DMA Router

User Application Logic (On-Chip Memories)

DMA Write

TX Slave

RX Master DMA Read

2, 6

UG-01097_sriov

2014.12.15

Figure 2-4: Steps To Perform a DMA Read

Understanding the DMA Functionality

2-7

The Read DMA operation includes the following steps:

1. The Descriptor Controller sends read descriptor instruction to initiate a DMA read.

2. The Descriptor Controller transmits a Memory Read TLP to the host starting at the source address.

3. The host returns DMA read data on the Avalon-ST interface.

4. The DMA Read Controller writes data to the destination address in the Application Layer memory.

5. The DMA Read module reports done status for each descriptor to the Descriptor Controller.

6. When all descriptors are complete, the Descriptor Controller sets the done bit of the last entry in the

descriptor table in host memory. The DMA Read Descriptor Controller sends this update to the TX Slave. The TX Slave drives the update to the Hard IP for PCI Express.

Getting Started with the SR-IOV DMA Example Design

Send Feedback

Altera Corporation

sriov_top_dma_gen3_x8_256.qsys

Stratix V Hard IP for PCI Express

SR-IOV Bridge

Rd_DC0 Rd_DC1 Rd_DC2 Rd_DC3

Read DMA Router

APPs - sriov_dma_app_g3x8_256b.qsys

rddc_ctl - rddc_ctl_256b.qsys wrdc_ctl - wrdc_ctl_256b.qsys

Wr_DC0 Wr_DC1 Wr_DC2 Wr_DC3

Write DMA Router

User Application Logic (On-Chip Memories)

DMA Write

TX Slave

RX Master DMA Read

2-8

Compiling the Example Design with the Quartus II Software

Figure 2-5: Steps To Perform a Write DMA

UG-01097_sriov

2014.12.15

The Write DMA operation includes the following steps:

1. The Descriptor Controller sends write descriptor instruction to initiate a DMA write.

2. The DMA Write reads data from the Application Layer memory.

3. Descriptor Controller transmits a Memory Write TLP to the host.

4. The DMA Write reports status for each descriptor to the Descriptor Controller.

5. When all descriptors are complete, the Descriptor Controller writes the ID of the last completed

descriptor to the EPLAST bit of the descriptor table.

Compiling the Example Design with the Quartus II Software

Complete the following steps to create and compile a Quartus II project.

1. In a terminal window, change to your working directory.

2. Copy the files from <install_dir>/ ip/altera/altera_pcie/altera_pcie_sriov/hw_devkit/ directory to your

working directory.

Altera Corporation

Getting Started with the SR-IOV DMA Example Design

Send Feedback

UG-01097_sriov

2014.12.15

Using the IP Catalog To Generate Your Stratix V Hard IP for PCI Express as a Separate

Component

These files specify Synopsys Design Constraints, Quartus II design constraints, and top-level connectivity.

3. On the Quartus II file menu, select the New Project Wizard. a. Specify top_hw for the project name.

b. To specify design constraints, on the Tools menu, select Tcl Scripts.

The Tcl Script dialog box appears.

c. Scroll down to select top.tcl. Click run.

The Quartus II software runs the design constraints.

4. On the Processing menu, select Start compilation. Quartus II compilation begins.

Using the IP Catalog To Generate Your Stratix V Hard IP for PCI Express as a Separate Component

You can also instantiate the Stratix V Hard IP for PCI Express IP Core as a separate component for integration into your project.

You can use the Quartus II IP Catalog and IP Parameter Editor to select, customize, and generate files representing your custom IP variation. The IP Catalog (Tools > IP Catalog) automatically displays IP cores available for your target device. Double-click any IP core name to launch the parameter editor and generate files representing your IP variation.

2-9

For more information about the customizing and generating IP Cores refer to Specifying IP Core Parameters and Options in Introduction to Altera IP Cores. For more information about upgrading older IP cores to the current release, refer to Upgrading Outdated IP Cores in Introduction to Altera IP Cores.

Note:

Your design must include the Transceiver Reconfiguration Controller IP Core and the Altera PCIe Reconfig Driver. Refer to the figure in the Qsys Design Flow section to learn how to connect this components.

Related Information

• Introduction to Altera IP Cores

• Managing Quartus II Projects

Getting Started with the SR-IOV DMA Example Design

Send Feedback

Altera Corporation

2014.12.15

www.altera.com

101 Innovation Drive, San Jose, CA 95134

Parameter Settings

UG-01097_sriov

Send Feedback

System Settings

Table 3-1: System Settings for PCI Express

Parameter Value Description

Lane Rate Gen1 (2.5 Gbps)

Gen2 (2.5/5.0 Gbps)

Gen3 (2.5/5.0/8.0

Gbps)

Number of Lanes ×1, ×2, ×4, ×8 Specifies the maximum number of lanes supported.

Port type Native Endpoint Specifies the port type. SR-IOV is only available for the Native

Specifies the maximum data rate at which the link can operate.

Endpoint in the current release. The Endpoint stores parameters in the Type 0 Configuration

Space.

PCI Express Base

2.1, 3.0 Select either the 2.1 or 3.0 specification.

Specification version

Application interface

Avalon-ST 256-bit Avalon-ST 128-bit

This core supports either a 128- and 256-bit Avalon-ST interface to the Application Layer.

ISO 9001:2008 Registered

3-2

System Settings

Parameter Value Description

UG-01097_sriov

2014.12.15

Reference clock frequency

RX Buffer credit allocation performance for received requests

100 MHz The PCI Express Base Specification 3.0 requires a

100 MHz ±300 ppm reference clock. The 125 MHz reference clock is provided as a convenience for systems that include a 125 MHz clock source. For more information about Gen3 operation, refer to 4.3.8 Refclk Specifications for 8.0 GT/sin the specification.

For Gen3, Altera recommends using a common reference clock (0 ppm). For designs with separate reference clocks (non 0 ppm), the PCS occasionally must insert SKP symbols, potentially causing the PCIe link to go to recovery. Gen1 and Gen2 modes are not affected by this issue. Systems using the common reference clock (0 ppm) are not affected by this issue. The primary repercussion of this is a slight decrease in bandwidth. On Gen3 x8 systems, this bandwidth impact is negligible. If non 0 ppm mode is required, so that separate reference clocks are being used, please contact Altera for further information and guidance.

Minimum

Low

Balanced

High

Maximum

Determines the allocation of posted header credits, posted data credits, non-posted header credits, completion header credits, and completion data credits in the 16 KByte RX buffer. The 5 settings allow you to adjust the credit allocation to optimize your system. The credit allocation for the selected setting displays in the message pane.

Refer to the Throughput Optimization chapter for more information about optimizing performance. The Flow Control chapter explains how the RX credit allocation and the

Maximum payload RX Buffer credit allocation and the Maximum payload size that you choose affect the allocation of flow control credits. You can set the Maximum payload size parameter on the Device tab.

Altera Corporation

The Message window dynamically updates the number of credits for Posted, Non-Posted Headers and Data, and Completion Headers and Data as you change this selection.

Parameter Settings

Send Feedback

UG-01097_sriov

2014.12.15

Parameter Value Description

• Minimum RX Buffer credit allocation—configures the

minimum PCIe specification allowed for non-posted and posted request credits, leaving most of the RX Buffer space for received completion header and data. Select this option for variations where application logic generates many read requests and only infrequently receives single requests from the PCIe link.

• Low—configures a slightly larger amount of RX Buffer

space for non-posted and posted request credits, but still dedicates most of the space for received completion header and data. Select this option for variations where application logic generates many read requests and infrequently receives small bursts of requests from the PCIe link. This option is recommended for typical endpoint applications where most of the PCIe traffic is generated by a DMA engine that is located in the endpoint application layer logic.

• Balanced—configures approximately half the RX Buffer

space to received requests and the other half of the RX Buffer space to received completions. Select this option for variations where the received requests and received completions are roughly equal.

• High—configures most of the RX Buffer space for received

requests and allocates a slightly larger than minimum amount of space for received completions. Select this option where most of the PCIe requests are generated by the other end of the PCIe link and the local application layer logic only infrequently generates a small burst of read requests. This option is recommended for typical root port applications where most of the PCIe traffic is generated by DMA engines located in the endpoints.

• Maximum—configures the minimum PCIe specification

allowed amount of completion space, leaving most of the RX Buffer space for received requests. Select this option when most of the PCIe requests are generated by the other end of the PCIe link and the local application layer logic never or only infrequently generates single read requests. This option is recommended for control and status endpoint applications that don't generate any PCIe requests of their own and only are the target of write and read requests from the root complex.

System Settings

3-3

Parameter Settings

Send Feedback

Altera Corporation

3-4

SR-IOV System Settings

Parameter Value Description

UG-01097_sriov

2014.12.15

Enable byte

On/Off When on, the RX and TX datapaths are parity protected. parity ports on Avalon-ST interface

Enable credit

On/Off When on, the core includes the tx_cons_cred_sel port. consumed selection port tx_ cons_cred_sel

Enable Hard IP

On/Off reset pulse at

power-up when using the soft reset controller

Related Information

PCI Express Base Specification 2.1 or 3.0

SR-IOV System Settings

Parity is odd. This parameter is only available for the Avalon-ST Stratix V

Hard IP for PCI Express.

When On, the soft reset controller generates a pulse at power up to reset the Hard IP. This pulse ensures that the Hard IP is reset after programming the device, regardless of the behavior of the dedicated PCI Express reset pin, perstn. This option is available for Gen2 and Gen3 designs that use a soft reset controller.

Parameter Value Description

Total active

1-2 This core supports 1 or 2 Physical Functions.

Physical Functions (PFs) :

Total Physical Function0 Virtual Functions (PF0 VFs):

Total Physical Function1 Virtual Functions (PF1 VFs):

System

0-128 Total number of VFs for PF0. From 0-7 PFs are supported

when ARI is not supported. From 4–128 VFs are supported when ARI is enabled. If PF1 is enabled, the sum of this field and PF1 VFs should not exceed 128. When ARI is enabled, the number of VFs should be a multiple of 4.

0-128 Total number of VFs for PF1. From 0-7 PFs are supported

4KB - 4MB Specifies the pages sizes supported. Supported Page Size:

Altera Corporation

Parameter Settings

Send Feedback

UG-01097_sriov

2014.12.15

Base Address Register (BAR) Settings

Parameter Value Description

3-5

Enable SR-IOV

On/Off

Turn this option on to include the SR-IOV functionality.

Support

Enable Alterna‐ tive Routing-ID (ARI) support

On/Off

This core supports the following configurations:

• 1 PF and 4-7 VFs with no ARI

• 1 PF and 4-128 VFs in multiples of 4 with ARI

• 2 PFs with 4-6 VFs and no ARI

• 2 PFs with 4-128 VFs in multiples of 4 with ARI Refer to Section 6.1.3 Alternative Routing-ID Interpretation

(ARI) of the PCI Express Base Specification more information about ARI.

Enable Functional Level

On/Off

When you turn this option on, each function can be individu‐ ally reset.

Reset (FLR)

Related Information

PCI Express Base Specification 2.1 or 3.0

Base Address Register (BAR) Settings

Each function can implement up to six BARs. You can configure up to six 32-bit BARs or three 64-bit BARs for both PFs and VFs. The BAR settings are the same for all VFs associated with a PF.

Table 3-2: BAR Registers

Parameter Value Description

Present Enabled/Disabled Indicates whether or not this BAR is instantiated.

Type 32-bit address

64-bit address

If you select 64-bit address, 2 contiguous BARs are combined to form a 64-bit BAR. you must set the higher numbered BAR to Disabled.

If the BAR TYPE of any even BAR is set to 64-bit memory, the next higher BAR supplies the upper address bits. The supported combinations for 64-bit BARs are {BAR1, BAR0}, {BAR3, BAR2}, {BAR4, BAR5}.

Parameter Settings

Send Feedback

Altera Corporation

3-6

Device Identification Registers

Parameter Value Description

UG-01097_sriov

2014.12.15

Prefetch‐ able

Prefetchable

Non-Prefetchable

Defining memory as prefetchable allows data in the region to be fetched ahead anticipating that the requestor may require more data from the same region than was originally requested. If you specify that a memory is prefetchable, it must have the following 2 attributes:

• Reads do not have side effects

• Write merging is allowed

Size 16 Bytes–2 GBytes Specifies the memory size.

Device Identification Registers

Table 3-3: Device ID Registers

The following table lists the default values of the read-only Device ID registers. You can use the parameter editor to change the values of these registers. At run time, you can change the values of these registers using the optional reconfiguration block signals. You can specify Device ID registers for each Physical Function.

Vendor ID 16 bits 0x00000000 Sets the read-only value of the Vendor ID register. This

parameter can not be set to 0xFFFF per the PCI Express Specification.

Address offset: 0x000.

Device ID 16 bits 0x00000000 Sets the read-only value of the Device ID register.

Address offset: 0x000.

Revision ID 8 bits 0x00000000 Sets the read-only value of the Revision ID register.

Address offset: 0x008.

Class code 24 bits 0x00000000 Sets the read-only value of the Class Code register.

Address offset: 0x008.

Subsystem Vendor ID

16 bits 0x00000000 Sets the read-only value of the Subsystem Vendor ID

register in the PCI Type 0 Configuration Space. This parameter cannot be set to 0xFFFF per the PCI Express Base Specification. This value is assigned by PCI-SIG to the device manufacturer.

Address offset: 0x02C.

Altera Corporation

Parameter Settings

Send Feedback

UG-01097_sriov

2014.12.15

Interrupt Capabilities

3-7

Subsystem Device ID

16 bits 0x00000000 Sets the read-only value of the Subsystem Device ID

Related Information

PCI Express Base Specification 2.1 or 3.0

Interrupt Capabilities

Table 3-4: MSI anad MSI-X Interrupt Settings

Each Physical Function defines its own MSI-X table settings. The VF MSI-X table settings are the same for all the Virtual Functions associated with each Physical Function.

Parameter Value Description

MSI Interrupt Settings PF0 MSI Requests 1,2,4,8,16,32 PF1 MSI Requests 1,2,4,8,16,32

Specifies the maximum number of MSI messages the Application

Layer can request. This value is reflected in Multiple Message

Capable field of the Message Control register, 0x050[31:16]. . For

MSI Interrupt Settings, if the PF MSI option is enabled, all PFs

support MSI capability.

MSI-X Interrupt Settings

PF MSI-X On/Off

When On, enables the MSI-X functionality. For PF and VF MSI-X Interrupt Settings, if PF MSI-X is enabled, all PFs

VF MSI-X On/Off

supports MSI-X capability.

Bit Range

MSI-X Table size [10:0] System software reads this field to determine the MSI-X Table

size <n>, which is encoded as <n–1>. For example, a returned value of 2047 indicates a table size of 2048. This field is readonly. Legal range is 0–2047 (211).

Address offset: 0x068[26:16]

MSI-X Table Offset

[31:0] Specifies the offset from the BAR indicated in theMSI-X Table

BAR Indicator. The lower 3 bits of the table BAR indicator

(BIR) are set to zero by software to form a 32-bit qwordaligned offset

(1)

. This field is read-only.

Parameter Settings

Send Feedback

Altera Corporation

3-8

PCI Express and PCI Capabilities Parameters

Parameter Value Description

UG-01097_sriov

2014.12.15

MSI-X Table BAR Indicator

MSI-X Pending Bit Array (PBA) Offset

MSI-X PBA BAR Indicator

PF0 Interrupt Pin

PF1 Interrupt Pin

[2:0] Specifies which one of a function’s BAR number. This field is

read-only. For 32-bit BARs, the legal range is 0–5. For 64-bit BARs, the legal range is 0, 2, or 4.

[31:0] Points to the MSI-X Pending Bit Array table. It is offset from

the BAR value indicated in MSI-X Table BAR Indicator. The lower 3 bits of the PBA BIR are set to zero by software to form a 32-bit qword-aligned offset. This field is read-only.

[2:0] Specifies which BAR number contains the MSI-X PBA. For

32-bit BARs, the legal range is 0–5. For 64-bit BARs, the legal range is 0, 2, or 4. This field is read-only.

Legacy Interrupts

inta–intd Applicable for PFs only to support legacy interrupts. When

enabled, the core receives interrupt indications from the

inta–intd

Application Layer on its INTA_IN, INTB_IN, INTC_IN and

INTD_IN inputs, and sends out Assert_INTx or Deassert_ INTx messages on the link in response to their activation or

deactivation, respectively. You can configure the Physical Functions with separate

interrupt pins. Or, both functions can share a common interrupt pin.

PF0 Interrupt Line

0-255

Defines the input to the interrupt controller (IRQ0 - IRQ15) in the Root Port that is activated by each Assert_INTx

PF1 Interrupt

0-255

message.

Line

Note:

1. Throughout this user guide, the terms word, dword and qword have the same meaning that they have

in the PCI Express Base Specification. A word is 16 bits, a dword is 32 bits, and a qword is 64 bits.

Related Information

PCI Express Base Specification Revision 2.1 or 3.0

PCI Express and PCI Capabilities Parameters

This group of parameters defines various capability properties of the IP core. Some of these parameters are stored in the PCI Configuration Space - PCI Compatible Configuration Space. The byte offset indicates the parameter address.

Altera Corporation

Parameter Settings

Send Feedback

UG-01097_sriov

2014.12.15

Device Capabilities

3-9

Parameter Possible

Values

Maximum

128 bytes

payload size

256 bytes

Completion timeout range

ABCD

BCD ABC

B A

None

Default

Value

Description

128 bytes Specifies the maximum payload size supported. This parameter

sets the read-only value of the max payload size supported field of the Device Capabilities register (0x084[2:0]). Address: 0x084.

ABCD Indicates device function support for the optional completion

timeout programmability mechanism. This mechanism allows system software to modify the completion timeout value. This field is applicable only to Root Ports and Endpoints that issue requests on their own behalf. Completion timeouts are specified and enabled in the Device Control 2 register (0x0A8) of the PCI Express Capability Structure Version. For all other functions this field is reserved and must be hardwired to 0x0000b. Four time value ranges are defined:

• Range A: 50 us to 10 ms

• Range B: 10 ms to 250 ms

• Range C: 250 ms to 4 s

• Range D: 4 s to 64 s Bits are set to show timeout value ranges supported. The

function must implement a timeout value in the range 50 s to 50 ms. The following values are used to specify the range:

Implement completion timeout disable

• None—Completion timeout programming is not supported

• 0001 Range A

• 0010 Range B

• 0011 Ranges A and B

• 0110 Ranges B and C

• 0111 Ranges A, B, and C

• 1110 Ranges B, C and D

• 1111 Ranges A, B, C, and D All other values are reserved. Altera recommends that the

completion timeout mechanism expire in no less than 10 ms.

On/Off On Disables the completion timeout mechanism. When On, the

core supports the completion timeout disable mechanism via the PCI Express Device Control Register 2. The Applica‐ tion Layer logic must implement the actual completion timeout mechanism for the required ranges. This option is forced to on for PCI Express version 2.0 and higher Endpoints.

Parameter Settings

Send Feedback

Altera Corporation

3-10

Error Reporting

UG-01097_sriov

2014.12.15

Parameter Possible

Values

Extended

On/Off On

tag support

Error Reporting

Parameter

Track Receive Completion Buffer Overflow

Error Reporting

Table 3-5: Error Reporting

Default

Value

Description

When enabled, the Application Layer supports up to 256 tags for non-posted requests. When disabled, the Application Layer supports up to 32 tags. The Hard IP with SR-IOV support disables tag checking. Consequently, the Application Layer must implement Completion tag checking.

Possible

Values

On/Off

You can use this status bit as an additional check to

Description

complement the soft logic that tracks space in the RX completion buffer. It is useful because the Endpoint RX Completion buffer must advertise infinite credits for RX Completions.

Parameter Value Default Value Description

Advanced error

On/Off Off When On, enables the Advanced Error Reporting (AER)

capability.

reporting (AER)

Enable ECRC checking

On/Off Off When On, enables ECRC checking. Sets the read-only

value of the ECRC check capable bit in the Advanced

Error Capabilities and Control Register. This

parameter requires you to enable the AER capability.

Enable ECRC generation

On/Off Off

When On, enables ECRC generation capability. Sets the read-only value of the ECRC generation capable bit in the Advanced Error Capabilities and Control

AER capability.

Enable ECRC forwarding on the Avalon-ST interface

On/Off Off When On, enables ECRC forwarding to the Application

Layer. On the Avalon-ST RX path, the incoming TLP contains the ECRC dword

(1)

and the TD bit is set if an ECRC exists. On the transmit the TLP from the Applica‐ tion Layer must contain the ECRC dword and have the

TD bit set.

Altera Corporation

Parameter Settings

Send Feedback

+ 128 hidden pages

Altera Stratix V Avalon-ST User Manual

Specifications and Main Features

Frequently Asked Questions

User Manual