AMD SR5690 Reference Manual

AMD SR5690 Databook

Technical Reference Manual

Rev 2.10

P/N: 43869_sr5690_ds_pub

Trademarks

AMD, the AMD Arrow logo, AMD PowerNow!, AMD Virtualization, AMD-V, and combinations thereof are trademarks of Advanced Micro Devices, Inc.

HyperTransport is a licensed trademark of the HyperTransport Technology Consortium.

Microsoft, Windows, and Windows Server are registered trademarks of Microsoft Corporation.

PCI Express and PCIe are registered trademarks of PCI-SIG.

Other product names used in this publication are for identification purposes only and may be trademarks of their respective companies.

Disclaimer

The contents of this document are provided in connection with Advanced Micro Devices, Inc. ("AMD") products. AMD makes no representations or warranties with respect to the accuracy or completeness of the contents of this publication and reserves the right to make changes to specifications and product descriptions at any time without notice. AMD assumes no liability whatsoever, and disclaims any express or implied warranty, relating to this document including, but not limited to, the implied warranty of merchantability, fitness for a particular purpose, or infringement of any intellectual property right. AMD shall not be liable for any damage, loss, expense, or claim of loss of any kind or character (including without limitation direct, indirect, consequential, exemplary, punitive, special, incidental or reliance damages) arising from use of or reliance on this document. No license, whether express, implied, arising by estoppel, or otherwise, to any intellectual property rights are granted by this publication. Except for AMD product purchased pursuant to AMD's Standard Terms and Conditions of Sale, and then only as express ly set forth therein, AMD's products are not designed, intended, authorized or warranted for use as components in systems intended for surgical implant into the body, or in other applications intended to support or sustain life, or in any other application in which the failure of AMD's product could create a situation where personal injury, death, or severe property or environmental damage may occur. AMD reserves the right to discontinue or make changes to its products at any time without notice.

Chapter 1: Overview

1.1 Introducing the SR5690 ......................................................................................................................................................1-1

1.2 SR5690 Features .................................................................................................................................................................1-1

1.2.1 CPU Interface .......................................................................................................................................................1-1

1.2.2 PCI Express® Interface ........................................................................................................................................1-1

1.2.3 A-Link Express II Interface..................................................................................................................................1-1

1.2.4 Multiple Processor Support ..................................................................................................................................1-2

1.2.5 Multiple Northbridge Support ..............................................................................................................................1-2

1.2.6 Power Management Features ...............................................................................................................................1-2

1.2.7 PC Design Guide Compliance..............................................................................................................................1-2

1.2.8 Test Capability Features .......................................................................................................................................1-2

1.2.9 Packaging .............................................................................................................................................................1-2

1.3 Software Features................................................................................................................................................................1-2

1.4 Device ID ............................................................................................................................................................................1-3

1.5 Branding Diagrams .............................................................................................................................................................1-3

1.6 Conventions and Notations .................................................................................................................................................1-4

1.6.1 Pin Names.............................................................................................................................................................1-4

1.6.2 Pin Types ..............................................................................................................................................................1-4

1.6.3 Numeric Representation .......................................................................................................................................1-4

1.6.4 Hyperlinks ............................................................................................................................................................1-5

1.6.5 Acronyms and Abbreviations ...............................................................................................................................1-5

Chapter 2: Functional Descriptions

2.1 HyperTransport™ Interface ................................................................................................................................................2-1

2.1.1 Overview ..............................................................................................................................................................2-1

2.1.2 HyperTransport™ Flow Control Buffers .............................................................................................................2-3

2.2 IOMMU ..............................................................................................................................................................................2-4

2.3 Multiple Northbridge Support.............................................................................................................................................2-4

2.4 Interrupt Handling...............................................................................................................................................................2-4

2.4.1 Legacy INTx Handling.........................................................................................................................................2-4

2.4.2 Non-SB IOAPIC Support .....................................................................................................................................2-4

2.4.3 Integrated IOAPIC Support..................................................................................................................................2-5

2.4.4 MSI Interrupt Handling and MSI to HT Interrupt Conversion ............................................................................2-5

2.4.5 Internally Generated Interrupts.............................................................................................................................2-5

2.4.6 IOMMU Interrupt Remapping .............................................................................................................................2-5

2.4.7 Interrupt Routing Architecture .............................................................................................................................2-5

2.5 RAS Features ......................................................................................................................................................................2-7

2.5.1 Parity Protection ...................................................................................................................................................2-7

2.5.2 SERR_FATAL# and NON_FATAL_CORR# Pins .............................................................................................2-7

2.5.3 NMI# and SYNCFLOODIN# ..............................................................................................................................2-8

2.5.4 Suggested Platform Level RAS Sideband Signal Connections............................................................................2-8

2.5.5 Error Reporting and Logging ...............................................................................................................................2-9

2.5.6 Interrupt Generation on Errors ........................................................................................................................... 2-11

2.5.7 Poisoned Data Support ....................................................................................................................................... 2-11

2.5.8 PCIe® Link Disable State ..................................................................................................................................2-11

2.5.9 HT Syncflood Based on PCIe® Error ............................................................................................................... 2-12

2.6 PCI Express® .................................................................................................................................................................. 2-12

2.6.1 PCIe® Ports ....................................................................................................................................................... 2-12

2.6.2 PCIe® Reset Signals.......................................................................................................................................... 2-12

2.6.3 PCIe® Hot-Pug.................................................................................................................................................. 2-13

2.7 External Clock Chip ......................................................................................................................................................... 2-14

Chapter 3: Pin Descriptions and Strap Options

3.1 Pin Assignment Top View ................................................................................................................................................. 3-2

3.2 SR5690 Interface Block Diagram ...................................................................................................................................... 3-4

3.3 CPU HyperTransport™ Interface ...................................................................................................................................... 3-4

3.4 PCI Express® Interfaces .................................................................................................................................................... 3-5

3.4.1 PCI Express® Interface for General Purpose External Devices ......................................................................... 3-5

3.4.2 A-Link Express II Interface to Southbridge ........................................................................................................ 3-5

3.4.3 Miscellaneous PCI Express® Signals ................................................................................................................. 3-6

3.5 Clock Interface ................................................................................................................................................................... 3-6

3.6 Power Management Pins.................................................................................................................................................... 3-7

3.7 Miscellaneous Pins ............................................................................................................................................................. 3-7

3.8 Power Pins.......................................................................................................................................................................... 3-8

3.9 Ground Pins........................................................................................................................................................................ 3-9

3.10 Strapping Options........................................................................................................................................................... 3-10

Chapter 4: Timing Specifications

4.1 HyperTransport™ Bus Timing .......................................................................................................................................... 4-1

4.2 PCI Express® Differential Clock AC Specifications......................................................................................................... 4-1

4.3 HyperTransport™ Reference Clock Timing Parameters ................................................................................................... 4-1

4.4 OSCIN Reference Clock Timing Parameters..................................................................................................................... 4-2

4.5 Power Rail Sequence.......................................................................................................................................................... 4-2

4.5.1 Power Up ............................................................................................................................................................. 4-3

4.5.2 Power Down ........................................................................................................................................................ 4-4

Chapter 5: Electrical Characteristics and Physical Data

5.1 Electrical Characteristics.................................................................................................................................................... 5-1

5.1.1 Maximum and Minimum Ratings........................................................................................................................ 5-1

5.1.2 DC Characteristics............................................................................................................................................... 5-1

5.2 SR5690 Thermal Characteristics........................................................................................................................................ 5-2

5.2.1 SR5690 Thermal Limits ...................................................................................................................................... 5-2

5.2.2 Thermal Diode Characteristics ............................................................................................................................ 5-3

5.3 Package Information .......................................................................................................................................................... 5-4

5.3.1 Pressure Specification.......................................................................................................................................... 5-5

5.3.2 Board Solder Reflow Process Recommendations ............................................................................................... 5-6

Chapter 6: Power Management and ACPI

6.1 ACPI Power Management Implementation ....................................................................................................................... 6-1

Chapter 7: Testability

7.1 Test Capability Features..................................................................................................................................................... 7-1

7.2 Test Interface...................................................................................................................................................................... 7-1

7.3 XOR Tree............................................................................................................................................................................7-1

7.3.1 Brief Description of an XOR Tree .......................................................................................................................7-1

7.3.2 Description of the XOR Tree for the SR5690 ......................................................................................................7-2

7.3.3 XOR Tree Activation ...........................................................................................................................................7-2

7.3.4 XOR Tree for the SR5690....................................................................................................................................7-3

7.4 VOH/VOL Test...................................................................................................................................................................7-4

7.4.1 Brief Description of a VOH/VOL Tree................................................................................................................7-4

7.4.2 VOH/VOL Tree Activation..................................................................................................................................7-5

7.4.3 VOH/VOL pin list ................................................................................................................................................7-6

Appendix A: Pin Listings

7.5 SR5690 Pin Listing Sorted by Ball Reference................................................................................................................... A-2

A.1 SR5690 Pin Listing Sorted by Pin Name .......................................................................................................................... A-9

Appendix B: Revision History

This page is left blank intentionally.

List of Figures

Figure 1-1: SR5690 Branding Diagram for A21 Production ASIC (Eutectic Part) ....................................................................... 1-3

Figure 1-2: SR5690 Branding Diagram for A21 Production ASIC (Lead Free Part) .................................................................... 1-3

Figure 1-3: SR5690 Alternate Branding for A21 Production ASIC (Lead Free Part) ................................................................... 1-4

Figure 2-1: SR5690 Internal Blocks and Interfaces ....................................................................................................................... 2-1

Figure 2-2: HyperTransport™ Interface Block Diagram ............................................................................................................... 2-2

Figure 2-3: SR5690 HyperTransport™ Interface Signals .............................................................................................................. 2-3

Figure 2-4: Interrupt Routing Paths in Legacy Mode .................................................................................................................... 2-6

Figure 2-5: Interrupt Routing Paths in Legacy Mode with Integrated IOAPIC ............................................................................. 2-6

Figure 2-6: Interrupt Routing Path in MSI Mode .......................................................................................................................... 2-7

Figure 2-7: Suggested Platform Level RAS Sideband Signal Connections ................................................................................... 2-9

Figure 2-8: Hot-plug Interface Connections ................................................................................................................................2-13

Figure 2-9: Hot-plug Signals between PCIe® Slot and I/O Expander ......................................................................................... 2-14

Figure 3-1: SR5690 Interface Block Diagram ............................................................................................................................... 3-4

Figure 4-1: SR5690 Power Rail Power Up Sequence .................................................................................................................... 4-3

Figure 5-2: SR5690 692-Pin FCBGA Package Outline ................................................................................................................. 5-4

Figure 5-3: SR5690 Ball Arrangement (Bottom View) ................................................................................................................. 5-5

Figure 5-4: RoHS/Lead-Free Solder (SAC305/405 Tin-Silver-Copper) Reflow Profile .............................................................. 5-7

Figure 7-1: XOR Tree .................................................................................................................................................................... 7-2

Figure 7-2: Sample of a Generic VOH/VOL Tree ......................................................................................................................... 7-5

This page is left blank intentionally.

List of Tables

Table 1-1: Device IDs for the SR5690/5670/5650 Chipset Family ................................................................................................1-3

Table 1-2: Pin Type Codes ..............................................................................................................................................................1-4

Table 1-3: Acronyms and Abbreviations ........................................................................................................................................1-5

Table 2-1: SR5690 HyperTransport™ Flow Control Buffers .........................................................................................................2-3

Table 2-2: Types of Errors Detectable by the SR5690 AER Implementation ..............................................................................2-10

Table 2-3: Types of HyperTransport™ Errors Supported by the SR5690 ....................................................................................2-11

Table 2-4: Possible Configurations for the PCI Express® General Purpose Links ......................................................................2-12

Table 2-5: GPP3a Ports with PCIe® Hot-Plug Support (Shaded) ................................................................................................2-14

Table 3-1: HyperTransport™ Interface ...........................................................................................................................................3-4

Table 3-2: PCI Express® Interface for General Purpose External Devices ....................................................................................3-5

Table 3-3: 1 x 4 Lane A-Link Express II Interface for Southbridge ...............................................................................................3-5

Table 3-4: Miscellaneous PCI Express® Signals ............................................................................................................................3-6

Table 3-5: Clock Interface ...............................................................................................................................................................3-6

Table 3-6: Power Management Pins ...............................................................................................................................................3-7

Table 3-7: Miscellaneous Pins ........................................................................................................................................................3-7

Table 3-8: Power Pins .....................................................................................................................................................................3-8

Table 3-9: Ground Pins ...................................................................................................................................................................3-9

Table 3-10: Strap Definitions for the SR5690 ..............................................................................................................................3-10

Table 3-11: Strap Definition for STRAP_PCIE_GPP_CFG .........................................................................................................3-10

Table 4-1: Timing Requirements for PCIe® Differential Clocks (GPP1_REFCLK, GPP2_REFCLK, and GPP3_REFCLK at

100MHz) .........................................................................................................................................................................................4-1

Table 4-2: Timing Requirements for HyperTransport™ Reference Clock (100MHz) ...................................................................4-1

Table 4-3: Timing Requirements for OSCIN Reference Clock (14.3181818MHz) .......................................................................4-2

Table 4-4: Power Rail Groupings for the SR5690 ..........................................................................................................................4-2

Table 4-5: SR5690 Power Rail Power-up Sequence .......................................................................................................................4-3

Table 5-1: Power Rail Maximum and Minimum Voltage Ratings .................................................................................................5-1

Table 5-2: Power Rail Current Ratings ...........................................................................................................................................5-1

Table 5-1: DC Characteristics for PCIe® Differential Clocks (GPP1_REFCLK, GPP2_REFCLK, and GPP3_REFCLK at 100MHz

) ........................................................................................................................................................................................................5-1

Table 5-3: DC Characteristics for 1.8V GPIO Pads ........................................................................................................................5-2

Table 5-4: DC Characteristics for the HyperTransport™ 100MHz Differential Clock (HT_REFCLK) .......................................5-2

Table 5-5: SR5690 Thermal Limits ................................................................................................................................................5-2

Table 5-6: SR5690 692-Pin FCBGA Package Physical Dimensions .............................................................................................5-4

Table 5-7: Recommended Board Solder Reflow Profile - RoHS/Lead-Free Solder ......................................................................5-6

Table 6-1: ACPI States Supported by the SR5690 .........................................................................................................................6-1

Table 7-1: Pins on the Test Interface ..............................................................................................................................................7-1

Table 7-2: Example of an XOR Tree ..............................................................................................................................................7-2

Table 7-3: SR5690 XOR Tree

Table 7-4: Truth Table for the VOH/VOL Tree Outputs ................................................................................................................7-5

Table 7-5: SR5690 VOH/VOL Tree ...............................................................................................................................................7-6

.........................................................................................................................................................7-3

This page intentionally left blank.

1.1 Introducing the SR5690

Chapter 1

Overview

The SR5690 (formerly RD890S) is the system logic of the latest server/workstation platform from AMD that enables its next generation CPUs. The SR5690 has a total of 46 PCI Express devices, and 4 are dedicated for the A-Link Express II interface to AMD’s Southbridges such as the SP5100 (formerly SB700S). The SR5690 also comes equipped with the new HyperTransport™ 3 and PCIe Gen 2 technologies. All of these are achieved by a highly integrated, thermally efficient design in a 29mm x 29mm package.

The SR5690 introduces a variety of Reliability, Availability and Serviceability (RAS) capabilities. These include parity protection for on-chip memories, PCI Express Advanced Error Reporting (AER), and advanced error handling capabilities for HyperTransport.

The SR5690 also supports a revision 1.26 compliant IOMMU (Input/Output Memory Management Unit) implementation for address translation and protection services. This feature allows virtual addresses from PCI Express endpoint devices to be translated to physical memory addresses. On-chip caching of address translations is provided to improve I/O performance. The device is also compliant with revision 1.0 of the PCI Express Address Translation Services (ATS) specification to enable ATS-compliant endpoint devices to cache address translation. These features enhance memory protection and support hardware-based I/O virtualization when combined with appropriate operating system or hypervisor software. Combined with AMD Virtualization™ (AMD-V™) technology, these features are designed to provide comprehensive platform level virtualization support.

1.2 SR5690 Features

1.2.1 CPU Interface

•

Supports 16-bit up/down HyperTransport™ (HT) 3.0 interface up to 5.2 GT/s.

• Supports 200, 400, 600, 800, and 1000 MHz HT1 frequencies.

• Supports 1200, 1400, 1600, 1800, 2000, 2200, 2400, and 2600 MHz HT3 frequencies (up to 2400 MHz only for the

RX980) .

• Supports “Shanghai” and subsequent series of AMD server/workstation and desktop processors through sockets F,

AM3, G34, and C32.

• Supports LDTSTOP interface and CPU throttling.

(PCIe®) lanes: 42 lanes are dedicated for external PCIe

1.2.2 PCI Express® Interface

•

Supports PCIe Gen 2 (version 2.0).

• Optimized peer-to-peer and general purpose link performance.

• Supports 42 PCIe Gen 2 general purpose lanes, and up to 11 devices on specific ports (possible configurations are

described in Section 2.6, “PCI

Express®”).

• Supports a revision 1.26 compliant IOMMU (Input/Output Memory Management Unit) implementation for address

translation and protection services. Please refer to the AMD I/O Virtualization Technology (IOMMU) Specification for more details.

• Supports PCIe hot-plug function for up to eight slots (firmware support required).

1.2.3 A-Link Express II Interface

One x4 A-Link Express II interface for connection to an AMD Southbridge. The A-Link Express II is a proprietary

•

interface developed by AMD based on the PCI Express technology, with additional Northbridge-Southbridge messaging functionalities.

1.2.4 Multiple Processor Support

•

Supports multiple-socket configurations for up to 8 processors on the same system.

1.2.5 Multiple Northbridge Support

Supports multiple-SR5690/5670/5650 configurations on the same system. See Section 2.3, “Multiple Northbridge

•

Support,” for details.

1.2.6 Power Management Features

•

Fully supports ACPI states S1, S3, S4, and S5.

• The Chip Power Management Support logic supports four device power states defined for the OnNow Architecture—

On, Standby, Suspend, and Off. Each power state can be achieved by software control bits.

• Support for AMD PowerNow!™ technology.

• Clocks are controlled dynamically using a mechanism that is transparent to the software. The ASIC hardware detects

idle blocks and turns off the clocks to those blocks in order to reduce power consumption.

• Supports dynamic lane reduction for the PCIe interfaces, adjusting to the task the number of lanes employed.

1.2.7 PC Design Guide Compliance

The SR5690 complies with all relevant Windows Logo Program (WLP) requirements from Microsoft® for WHQL certification.

Software Features

1.2.8 Test Capability Features

The SR5690 has a variety of test modes and capabilities that provide a very high fault coverage and low DPM (Defect Per Million) ratio:

• Full scan implementation on the digital core logic which provides about 97% fault coverage through ATPG

(Automatic Test Pattern Generation Vectors).

• Dedicated test logic for the on-chip custom memory macros to provide complete coverage on these modules.

• A JTAG test mode in order to allow board level testing of neighboring devices.

• An XOR tree test mode on all the digital I/O's to allow for proper soldering verification at the board level.

• Access to the analog modules and PLLs in the SR5690 in order to allow full evaluation and characterization of these

modules.

• IDDQ mode support to allow chip evaluation through current leakage measurements.

• Highly advanced signal observability through the debug port.

These test modes can be accessed through the settings of the instruction register of the JTAG circuitry.

1.2.9 Packaging

•

Single chip solution in 65nm, 1.1V CMOS technology.

• Flip chip design in a 29mm x 29mm 692-FCBGA package.

1.3 Software Features

• Supports Windows Server

• Supports corporate manageability requirements such as DMI.

• ACPI support.

• Full write combining support for maximum performance.

• Comprehensive OS and API support.

• Extensive Power Management support.

2003, Windows Server® 2008, Red Hat Enterprise Linux, SUSE Linux, and Solaris.

Device ID

Northbridge YYWW MADE IN TAIWAN WXXXXX 215-0716022

* YY - Assembly Start Year WW - Assembly Start Week

Part Number

Date Code*

AMD Product Type

AMD Logo

Wafer Lot Number

Country of Origin

Note: Branding can be in laser, ink, or mixed laser-and-ink marking.

Northbridge YYWW MADE IN TAIWAN WXXXXX 215-0716038

* YY - Assembly Start Year WW - Assembly Start Week

Part Number

Date Code*

AMD Product Type

AMD Logo

Wafer Lot Number

Country of Origin

Note: Branding can be in laser, ink, or mixed laser-and-ink marking.

1.4 Device ID

The SR5690 is a member of the AMD chipset family, which consists of different devices designed to support different platforms. Each device is identified by a device ID, which is stored in the NB_DEVICE_ID register. The device IDs for the SR5650/5690/5670 chipset family are as follows:

Table 1-1 Device IDs for the SR5690/5670/5650 Chipset Family

Device Device ID

SR5690 5A10h

SR5670 5A12h

SR5650 5A13h

1.5 Branding Diagrams

Figure 1-1 SR5690 Branding Diagram for A21 Production ASIC (Eutectic Part)

Figure 1-2 SR5690 Branding Diagram for A21 Production ASIC (Lead Free Part)

Figure 1-3 SR5690 Alternate Branding for A21 Production ASIC (Lead Free Part)

Northbridge YYWW MADE IN TAIWAN WXXXXX 215-0716056

* YY - Assembly Start Year WW - Assembly Start Week

Part Number

Date Code*

AMD Product Type

AMD Logo

Wafer Lot Number

Country of Origin

Note: Branding can be in laser, ink, or mixed laser-and-ink marking.

1.6 Conventions and Notations

Conventions and Notations

The following sections explain the conventions used throughout this manual.

1.6.1 Pin Names

Pins are identified by their pin names or ball references. All active-low signals are identified by the suffix ‘#’ in their names (e.g., SYSRESET#).

1.6.2 Pin Types

The pins are assigned different codes according to their operational characteristics. These codes are listed in Table 1-2.

Table 1-2 Pin Type Codes

Code Pin Type

I Digital Input

O Digital Output

I/O Bi-Directional Digital Input or Output

M Multifunctional

Pwr Power

Gnd Ground

A-O Analog Output

A-I Analog Input

A-I/O Analog Bi-Directional Input/Output

A-Pwr Analog Power

A-Gnd Analog Ground

Other Pin types not included in any of the categories above

1.6.3 Numeric Representation

Hexadecimal numbers are appended with “h” whenever there is a risk of ambiguity. Other numbers are in decimal.

Pins of identical functions but different trailing digits (e.g., DFT_GPIO0, DFT_GPIO1, ...DFT_GPIO5) are referred to collectively by specifying their digits in square brackets and with colons (i.e., “DFT_GPIO[5:0]”). A similar short-hand

Conventions and Notations

notation is used to indicate bit occupation in a register. For example, NB_COMMAND[15:10] refers to the bit positions 10 through 15 of the NB_COMMAND register.

1.6.4 Hyperlinks

Phrases or sentences in blue italic font are hyperlinks to other parts of the manual. Users of the PDF version of this manual can click on the links to go directly to the referenced sections, tables, or figures.

1.6.5 Acronyms and Abbreviations

The following is a list of the acronyms and abbreviations used in this manual.

Table 1-3 Acronyms and Abbreviations

Acronym Full Expression

ACPI Advanced Configuration and Power Interface

ASPM Active State Power Management

A-Link-E A-Link Express interface between the Northbridge and Southbridge.

BGA Ball Grid Array

BIOS

BIST Built In Self Test.

DBI Dynamic Bus Inversion

DPM Defects per Million

EPROM Erasable Programmable Read Only Memory

FCBGA Flip Chip Ball Grid Array

FIFO First In, First Out

VSS Ground

GPIO General Purpose Input/Output

HT HyperTransport™ interface

IDDQ Direct Drain Quiescent Current

IOMMU Input/Output Memory Management Unit

JTAG Joint Test Access Group. An IEEE standard.

MB Mega Byte

NB Northbridge

PCI Peripheral Component Interface

PCIe

PLL Phase Locked Loop

POST Power On Self Test

PD Pull-down Resistor

PU Pull-up Resistor

RAS Reliability, Availability and Serviceability

SB Southbridge

TBA To Be Added

VRM Voltage Regulation Module

Basic Input Output System. Initialization code stored in a ROM or Flash RAM used to start up a system or expansion card.

PCI Express

This page is left blank intentionally.

Conventions and Notations

Chapter 2

HyperTransport™ 3

Unit

CPU

Interface

Root

CPU

Southbridge

Complex

A-Link-E

Interface

GPP3 Interface

PCIe

(6 Lanes for 6 ports, plus

Expansion

Slots

(1 x 4 Lanes)

IO Controller

GPP1 Interface

PCIe

GPP2 Interface

PCIe

(1 x 16 or 2 x 8 Lanes) (1 x 16 or 2 x 8 Lanes)

IOMMU

4 Lanes for 1 port)

Expansion

Slots

Expansion

Slots

Functional Descriptions

This chapter describes the functional operation of the major interfaces of the SR5690 system logic chip. Figure 2-1 illustrates the SR5690 internal blocks and interfaces.

Figure 2-1 SR5690 Internal Blocks and Interfaces

2.1 HyperTransport™ Interface

2.1.1 Overview

The SR5690 is optimized to interface with “Shanghai” and subsequent series of AMD server/workstation and desktop processors through sockets F, AM3, G34, and C32. The SR5690 supports HyperTransport HyperTransport 1 (HT1) for backward compatibility and for initial boot-up. For a detailed description of the interface, please refer to the HyperTransport I/O Link Specification from the HyperTransport Consortium. Figure 2-2,

“HyperTransport™ Interface Block Diagram,” illustrates the basic blocks of the host bus interface of the SR5690.

™

3 (HT3), as well as

I/O Controller

Upstream Arbitration

IOMMU L2

10.4 GB/s to CPU

Response

Interface

Host Interface

Host

requests

Host read

responses

Host read

responses

DMA requests

DMA

read

response

data

IOMMU

requests

10.4 GB/s from CPU

Rx PHYTx PHY

Rx PHY InterfaceTx PHY Interface

Protocol ReceiverProtocol Transmitter

PCIe® Cores

HyperTransport™ Interface

Figure 2-2 HyperTransport™ Interface Block Diagram

The SR5690 HyperTransport bus interface consists of 16 unidirectional differential Command/Address/Data pins, and 2 differential Control pins and 2 differential Clock pins in both the upstream and downstream directions. On power up, the link is 8-bit wide and runs at a default speed of 400MT/s in HyperTransport 1 mode. After negotiation, carried out by the HW and SW together, the link width can be brought up to the full 16-bit width and the interface can run up to 5.2GT/s in HyperTransport 3 mode. In HyperTransport 1 mode, the interface operates by clock-forwarding while in HyperTransport 3 mode, the interface operates by dynamic phase recovery, with frequency information propagated over the clock pins. The interface is illustrated below in Figure 2-3, “SR5690 HyperTransport™ Interface Signals.” The signal name and direction for each signal is shown with respect to the SR5690. Detailed descriptions of the signals are given in Section 3.3,

“CPU HyperTransport™ Interface‚’ on page 3-4.

HyperTransport™ Interface

HT_TXCALP

HT_RXCALN

HT_RXCALP

HT_TXCALN

HT_RXCADN

SR5690

CPU

HT_RXCADP

HT_RXCTLN

HT_RXCTLP

HT_RXCLKN

HT_RXCLKP

HT_TXCADN

HT_TXCADP

HT_TXCTLN

HT_TXCTLP

HT_TXCLKN

HT_TXCLKP

2.1.2 HyperTransport™ Flow Control Buffers

Figure 2-3 SR5690 HyperTransport™ Interface Signals

The SR5690 HyperTransport interface has the following features:

• HyperTransport 3.0 compliant

• 16-bit and 8-bit link widths supported. Width for each direction of the link is independently controlled.

• 400MT/s to 5.2GT/s link speeds in increments of 400MT/s (up to 2GT/s only for HyperTransport 1 mode)

• DC-coupled HyperTransport mode only

• UnitID clumping for x16 PCI Express

ports

• Isochronous flow-control mode for Southbridge audio and IOMMU traffic

• 64-bit address extension support (52-bit physical addressing)

• Link disconnection with tristate, LS1, and LS2 low-power modes

• Error retry in HyperTransport 3 mode

• Full HyperTransport-defined BIST support for both internal and external loopback modes

The SR5690 HTIU implements the following flow control buffers in its receiver:

Table 2-1 SR5690 HyperTransport™ Flow Control Buffers

Flow Control Buffer Type Posted Non-Posted Response

Cmd 16 16 Advertise 63 credits.

Data 16 1 Advertise 63 credits.

ISOC Cmd 0 0 Advertise 63 credits.

ISOC Data 0 0 Advertise 63 credits.

2.2 IOMMU

The SR5690’s IOMMU (Input/Output Memory Management Unit) block provides address translation and protection services as described in version 1.26 of the AMD I/O Virtualization Technology (IOMMU) Specification. The SR5690 also supports the PCI Express Address Translation Services 1.0 Specification, which allows the supporting of endpoint devices to request and cache address translations.

When DMA requests containing virtual addresses are received, the IOMMU looks up the page translation tables located in the system memory in order to convert the virtual addresses into physical addresses and to verify access privileges. On-chip caching is provided in order to speed up translation and reduce or eliminate the number of system memory accesses required. Every PCIe core contains a local translation cache, and the SR5690 also contains a shared global translation cache.

The SR5690 supports up to 2 52-bit physical address space.

domains, each of which can utilize a separate 64-bit virtual address space. It supports a

2.3 Multiple Northbridge Support

Multiple SR5690/5670/5650 (referred to as “SR56x0”below) Northbridges may be implemented in the same system given enough free HyperTransport links from the processor complex. However, only a single Southbridge may be used. The SR56x0 attached to the Southbridge is called the primary SR56x0, and any other instance of SR56x0 is called a secondary SR56x0. The A-Link Express interface on any secondary SR56x0 must be left unconnected, and it cannot be used to support any PCI Express endpoint devices.

IOMMU

The PWM_GPIO5 pin-strap is used to indicate whether an SR56x0 is a primary or a secondary Northbridge. If no pull-down resistor is attached on the pin, the internal pull-up resistor on it will set the strap value to “1,” indicating the device to be a primary Northbridge. On any secondary SR56x0, the PWM_GPIO5 pin-strap must be pulled low.

In the multi-NB mode, special PCI Express messages for functions such as PME may be passed from a secondary SR56x0 to the primary SR56x0 or the Southbridge over the HyperTransport bus. If the SR56x0’s internal IOAPIC is not used, INTx messages may also be forwarded over the HyperTransport bus to the Southbridge IOAPIC. Peer-to-peer writes between PCI Express endpoints are also allowed between any SR56x0 and another by routing peer-to-peer requests over the HyperTransport bus.

Note: As it is possible to mix-and-match SR5650, SR5670, and SR5690 on the same system, whenever a multiple-SR5690 configuration is being referred to in this document, it actually represents any combination of SR5650, SR5670, and SR5690 possible under that situation. Some constrains may apply.

2.4 Interrupt Handling

2.4.1 Legacy INTx Handling

In legacy interrupt mode, all INTx messages must be routed to the Southbridge IOAPIC. The primary NB directs all INTx messages directly down to the Southbridge IOAPIC. Secondary NBs direct INTx messages up to the processor complex, where they are broadcast down to all HT devices. See Section 2.3, “Multiple Northbridge Support‚’ on page 2-4 for details.

The 4 legacy interrupts sent by endpoint devices (INT A/B/C/D) may undergo a 2-stage programmable swizzling process that maps them onto the 8 possible internal INTx messages (INT A/B/C/D/E/F/G/H). The first swizzling stage is performed by rotating the interrupt message number based upon the bridge device number. The second stage is register controllable on a per-bridge basis and maps the rotated INT A/B/C/D onto INT E/F/G/H. INT A to H messages sent to the Southbridge are mapped onto the SB IOAPIC interrupt redirection table entries 16 to 23.

2.4.2 Non-SB IOAPIC Support

The SR5690 supports routing legacy IOAPIC memory-mapped I/O addresses (0xFECx_xxxx) to any PCI Express port to support endpoint devices with integrated IOAPIC.

Interrupt Handling

2.4.3 Integrated IOAPIC Support

The SR5690 supports routing local INTx messages to its integrated IOAPIC. The integrated IOAPIC contains a 32-entry redirection table. INTx messages from endpoint devices, bridges, HTIU, and IOMMU can be mapped onto different redirection entries under register control.

2.4.4 MSI Interrupt Handling and MSI to HT Interrupt Conversion

In MSI interrupt mode, all interrupts are sent directly from the endpoint devices through the SR5690 up to the processor complex. All MSI interrupts are converted into HT-formatted interrupts. For MSIs from PCI Express endpoint devices and internally generated PCI Express interrupts, the conversion occurs in the associated IOMMU L1 block. For IOMMU interrupts and, optionally, HT error interrupts and internal parity error interrupts, the conversion occurs in the HTIU block. HT error interrupts and internal parity error interrupts may be optionally redirected to an MSI generation block underneath the SB VC1 IOMMU L1 so that they can be remapped by IOMMU. IOMMU internal MSI interrupts are never remapped.

The PCI configuration spaces of each on-board device contains a fixed HT MSI mapping capability (except for Device 1, which is unused). This implies that all MSI interrupts with address 0xFEEx_xxxx have to be converted to HT interrupts. Because of this, software is required to program all MSI address registers with an 0xFEEx_xxxx address.

2.4.5 Internally Generated Interrupts

The SR5690 may internally generate interrupts for the following purposes:

• PCI Express error

• PCI Express PME

• HT error

• Internal parity error

• IOMMU command handler

• IOMMU event logger

Internally generated interrupts may be in either legacy INTx or MSI format. Internal MSI interrupt sources do not support per-vector masking.

2.4.6 IOMMU Interrupt Remapping

When the IOMMU is enabled, interrupts generated downstream of the IOMMU are remapped based upon the IOMMU tables. The following classes of interrupts are not remapped by the IOMMU because they are generated upstream of the IOMMU:

• HT error (optional)

• Internal parity error (optional)

• IOMMU command handler and event logger

2.4.7 Interrupt Routing Architecture

2.4.7.1 Legacy Mode

Primary SR5690: Legacy INTx messages are routed directly to the SB IOAPIC. The SB IOAPIC generates upstream

interrupt requests, which are translated by the IOMMU before they are delivered up to the processor complex.

Secondary SR5690: Legacy INTx messages are routed over HyperTransport through the processor complex to the primary SR5690, which forwards them to the SB IOAPIC. The SB IOAPIC generates upstream interrupt requests, which are translated by the IOMMU before being delivered up to the processor complex.

The routing paths are illustrated in Figure 2-4 below.

CPU

SR5690

CPU

SR5690

PCI-E

Endpoint

Device

PCI-E

Endpoint

Device

INTx Message from device attached to primary SR5690

INTx Message from device attached to secondary SR5690

Interrupts from SB IOAPIC

CPU

PCI-E

Endpoint

Device

IOMMU

PCI-

Express

IOAPIC

SR5690

INTx Message from PCI-Express device attached to SR5690

Internal interrupt

Remapped HT Interrupt

SR5690

PCI-E

Endpoint

Device

CPU

Figure 2-4 Interrupt Routing Paths in Legacy Mode

2.4.7.2 Legacy Mode with Integrated IOAPIC

For both the primary and secondary SR5690s, legacy INTx messages are routed to the integrated IOAPICs of the SR5690s, which generates interrupt requests. These requests are remapped by the IOMMU before being delivered up to the processor complex. If an INTx message gets directed to an IOAPIC table entry that is not enabled, the IOAPIC sends the INTx message back to the IOC to go to the SB PIC/IOAPIC.

Interrupt Handling

The routing paths are illustrated in Figure 2-5 below.

Figure 2-5 Interrupt Routing Paths in Legacy Mode with Integrated IOAPIC

RAS Features

CPU

PCI-E

Endpoint

Device

IOMMU

PCI-

Express

SR5690

MSI Interrupt from PCI-Express device attached to SR5690

Remapped HT Interrupt

SR5690

PCI-E

Endpoint

Device

CPU

2.4.7.3 MSI Mode

For both the primary and secondary SR5690s: MSI interrupt requests are remapped by the IOMMU and sent up to the

processor complex. The routing path is illustrated in Figure 2-6 below.

2.5 RAS Features

2.5.1 Parity Protection

All memories in SR5690 are parity protected to reduce the possibility of silent data corruption. Multiple parity words are interleaved to convert burst errors (multiple physically adjacent bits corrupted) into multiple single-bit detectable errors to increase robustness. The minimum number of interleaved parity words in any on-board memory is 4. All macros contain test circuitry for software to generate false errors on either the read or write side of the memory for verification of error handling routines. Error injection circuitry only corrupts parity bits rather than real data bits to avoid data corruption.

2.5.1.1 Parity Protection for IOMMU Cache Memories

2.5.1.2 Parity Protection for Normal Memories

2.5.2 SERR_FATAL# and NON_FATAL_CORR# Pins

All IOMMU cache memories are parity protected. When a parity error is detected, the access from the associated bank is marked as an automatic miss. The cache line is marked as invalid and may later be overwritten with data from system memory (which is ECC protected). The error is logged in a status bit and an optional interrupt is generated (either fatal, non-fatal, or correctable parity error).

All normal memories are also parity protected. When a parity error is detected, the failure is likely to be fatal as there is no automatic recovery mechanism and no way for hardware to tag a specific request or operation with the error. The error is logged in a status bit for later diagnosis and an optional interrupt is generated (either fatal or non-fatal parity error).

The SR5690 implements a dedicated pin, DBG_GPIO0/SERR_FATAL#, to signal either a system or a fatal error, which can be used to signal a BMC for further actions. SERR_FATAL# may be asserted on various error conditions like HT syncflood, as well as internal parity errors or fatal errors for which signalling by SERR_FATAL# is enabled. Fatal errors are identified via the fatal error status bits.

Figure 2-6 Interrupt Routing Path in MSI Mode

Non-fatal or correctable errors may be likewise signalled via DBG_GPIO3/NON_FATAL_CORR#.

The SERR_FATAL# and NON_FATAL_CORR# pin functionalities are disabled on warm reset.

2.5.3 NMI# and SYNCFLOODIN#

The SR5690 may configure the DFT_GPIO0/NMI# pin as an input pin for triggering an upstream NMI packet to the processor complex. The pin should be driven by a BMC. An internal sticky status bit records the use of the NMI# pin.

Also, the SR5690 may configure the DFT_GPIO5/SYNCFLOODIN# pin as an input pin for triggering a HyperTransport syncflood event. The pin should driven by a BMC. An internal sticky status bit records the use of the SYNCFLOODIN# pin.

2.5.4 Suggested Platform Level RAS Sideband Signal Connections

Figure 2-7 is a logical diagram showing suggestions for RAS sideband signal connections at the platform level .

RAS Features

+ 56 hidden pages

AMD SR5690 Reference Manual

Table of Contents

List of Figures

List of Tables

1.1 Introducing the SR5690

Overview

1.2 SR5690 Features

1.2.1 CPU Interface

1.2.2 PCI Express® Interface

1.2.3 A-Link Express II Interface

1.2.4 Multiple Processor Support

1.2.5 Multiple Northbridge Support

1.2.6 Power Management Features

1.2.7 PC Design Guide Compliance

1.2.8 Test Capability Features

1.2.9 Packaging

1.3 Software Features

1.4 Device ID

Table 1-1 Device IDs for the SR5690/5670/5650 Chipset Family

1.5 Branding Diagrams

1.6 Conventions and Notations

1.6.1 Pin Names

1.6.2 Pin Types

Table 1-2 Pin Type Codes

1.6.3 Numeric Representation

1.6.4 Hyperlinks

1.6.5 Acronyms and Abbreviations

Table 1-3 Acronyms and Abbreviations

Functional Descriptions

2.1 HyperTransport™ Interface

2.1.1 Overview

2.1.2 HyperTransport™ Flow Control Buffers

Table 2-1 SR5690 HyperTransport™ Flow Control Buffers

2.2 IOMMU

2.3 Multiple Northbridge Support

2.4 Interrupt Handling

2.4.1 Legacy INTx Handling

2.4.2 Non-SB IOAPIC Support

2.4.3 Integrated IOAPIC Support

2.4.4 MSI Interrupt Handling and MSI to HT Interrupt Conversion

2.4.5 Internally Generated Interrupts

2.4.6 IOMMU Interrupt Remapping

2.4.7 Interrupt Routing Architecture

2.4.7.1 Legacy Mode

2.4.7.2 Legacy Mode with Integrated IOAPIC

2.4.7.3 MSI Mode

2.5 RAS Features

2.5.1 Parity Protection

2.5.1.1 Parity Protection for IOMMU Cache Memories

2.5.1.2 Parity Protection for Normal Memories

2.5.2 SERR_FATAL# and NON_FATAL_CORR# Pins

2.5.3 NMI# and SYNCFLOODIN#

2.5.4 Suggested Platform Level RAS Sideband Signal Connections