Integrated Device Technology, Inc. reserves the right to make changes to its products or specifications at any time, without notice, in order to improve design or performance
and to supply the best possible product. IDT does not assume any responsibility for use of any circuitry described other than the circuitry embodied in an IDT product. The
Company makes no representations that circuitry described herein is free from patent infringement or other rights of third parties which may result from its use. No license is
granted by implication or otherwise under any patent, patent rights or other rights, of Integrated Device Technology, Inc.
DISCLAIMER
Boards that fail to function should be returned to IDT for replacement. Credit will not be given for the failed boards nor will a
Failure Analysis be performed.
Integrated Device Technology's products are not authorized for use as critical components in life support devices or systems unless a specific written agreement pertaining to
such intended use is executed between the manufacturer and an officer of IDT.
1. Life support devices or systems are devices or systems which (a) are intended for surgical implant into the body or (b) support or sustain life and whose failure to perform,
when properly used in accordance with instructions for use provided in the labeling, can be reasonably expected to result in a significant injury to the user.
2. A critical component is any components of a life support device or system whose failure to perform can be reasonably expected to cause the failure of the life support device
or system, or to affect its safety or effectiveness.
IDT, the IDT logo, and Integrated Device Technology are trademarks or registered trademarks of Integrated Device Technology, Inc.
The 89HPES24NT24G2 switch is a member of the IDT PCI Express® Inter-Domain Switch family of
products. It is a PCIe® Base Specification 2.1 compliant (Gen2) 24-lane, 24-port switch. The EB-LOGAN19 Evaluation Board provides an evaluation platform for the PES24NT24G2 switch and for other members
of this switch family including PES16NT16G2 and PES12NT12G2.
Detailed information related to configuration of number of ports and lanes in the switch device can be
found in the Device User Manual and the Device Datasheet. The evaluation board, along with additional
adapters and daughter boards provided by IDT, can be configured to test every possible combination of the
number of lanes and ports offered by the switch. Advanced capabilities such as switch partitioning, NTB,
DMA and local port clocking can be evaluated with the evaluation board.
The EB-LOGAN-19 brings out all 24 lanes of the device to two Mezzanine connectors and two SAS
connectors (see
of daughter cards (provided by IDT) can then be plugged into the Mezzanine connectors to facilitate
connectivity to one x8 or two x4 or four x2 or eight x1 link partners. Link partners may be plugged directly
into these daughter cards or they can be connected to these daughter cards via SAS or SATA cables and a
different board with PCIe slots known as the 12-PACK board (provided by IDT). Given that majority of the
hosts / servers offer PCIe standard slots, IDT provides the necessary adapter cards that may be plugged
into these host / server slots as well as the cables that connect such adapters to the daughter cards which
in turn are plugged into the main evaluation board on which the IDT PCIe switch device is populated.
The EB-LOGAN-19 is also used by IDT to reproduce system-level hardware or software issues reported
by customers.
LOGAN-19 board.
Figure 1.1) located close to the device - one connector per stack of 4 lanes. Various types
Figure 1.1 illustrates the functional block diagram representing the main parts of the EB-
Figure 1.1 Function Block Diagram of the EB-LOGAN-19 Evaluation Board
IDT Description of the EB-LOGAN-19 Evaluation Board
Notes
Board Features
Hardware
PES24NT24G2 PCIe 24-port switch
– Twenty four ports (each x1) - for port 8 and higher, adjacent ports may be combined to create x2,
x4 or x8 ports
– PCIe Base Specification Revision 2.1 compliant (Gen2 SerDes speeds of 5 GT/S)
– Up to 2048 byte maximum Payload Size
– Automatic lane reversal and polarity inversion supported on all lanes
– Automatic per port link width negotiation to x8, x4, x2, x1
– Power on reconfiguration via optional serial EEPROM connected to the SMBUS Master interface
Upstream, Downstream Ports
– The EB-LOGAN-19 has minimum of one port configured as upstream port to be plugged into a
host slot through an adaptor and a cable.
– Up to 23 ports can be configured as downstream ports, for PCIe endpoint add-on cards to be
plugged in. The slot connectors can be configured to be x1, x2, x4 or x8, but are mechanically
open-ended on one side to allow card widths greater than x8 (e.g. x16) to be populated.
– When used in multi-partition mode, the device can be programmed through the serial EEPROM
to generate the appropriate number of upstream and downstream ports per partition.
Numerous user selectable configurations set using onboard jumpers and DIP-switches
– Source of clock - host clock or onboard clock generator
– Two clock rates (100/125 MHz) from an onboard clock generator
– Flexible clocking modes
• Common clock
• Non-common clock
• Local port clocking on ports that support this feature
– Boot mode selection
SMBUS Slave Interface (4 pin header)
SMBUS Master Interface connected to the Serial EEPROMs through I/O expander
Push button for Warm Reset
Many LEDs to display status, reset, power, hotplug, etc.
JTAG connector to the PES24NT24G2 JTAG pins.
Software
There is no software or firmware executed on the board. However, useful software is provided along
with the Evaluation Board to facilitate configuration and evaluation of the PES24NT24G2 within host
systems running popular operating systems.
Installation programs
– Operating Systems Supported: WindowsServer200x, WindowsXP, Vista, Linux
GUI based application for Windows and Linux
– Allows users to view and modify registers in the PES24NT24G2
– Binary file generator for programming the serial EEPROMs attached to the SMBUS.
Other
SMBUS cable may be required for certain evaluation exercises.
SMA connectors are provided on the EB-LOGAN-19 board for clock outputs.
Revision History
April 13, 2010: Initial publication of evaluation board manual.
This chapter discusses the steps required to configure and install the EB-LOGAN-19 evaluation board.
All available DIP switches and jumper configurations are explained in detail.
The primary installation steps are:
1. Configure jumper/switch options suitable for the evaluation or application requirements.
2. Connect PCI Express endpoint cards to the downstream port PCIe slots on the evaluation board.
3. Make sure that the host system (motherboard with root complex chipset) is powered off.
4. Connect the evaluation board to the host system.
5. Apply power to the host system.
The EB-LOGAN-19 board is typically shipped with all jumpers and switches configured to their default
settings. In most cases, the board does not require further modification or setup however please visit IDT
website and fill out the Technical Support Request form at http://www.idt.com/?app=TechSupport for other
configurations.
PCI Express Mezzanine and Edge Adapters
The PCI Express lanes are broken out to four Mezzanine connectors on EB-LOGAN-19 Evaluation
Board. The adapter cards are used to convert Mezzanine connector into PCI Express slot connector(s) or
Internal mini SAS (iSAS) connectors or both. A Bifurcated Mezzanine Card has two mechanical x8 PCIe
Slots (x4 electrically) while a Merged Mezzanine Card has single x8 PCIe Slot. Pictured in
IDT Installation of the EB-LOGAN-19 Evaluation Board
Notes
Pictured in Figure 2.2 is the mini-SAS Mezzanine card which consists of two iSAS and two SATA
connectors. Each iSAS connector supports up to PCI Express x4 width and the SATA connectors are used
for clock and reset signals of each x4 or less stack/port. An iSAS-to-SATA breakout cable shown in
Figure
2.3 is used connect from iSAS to edge adapter and/or 12PACK.
The PCI Express Edge to SATA Adapter, pictured in Figure 2.4, can be inserted into any physical PCIe
slot on a host system and in combination with mini-SAS Mezzanine Card, such as the one in Figure 2.2, to
form a link between evaluation main board and the host system. There are 5 SATA connectors which one
connector (J7) is for clock and reset, and the rest supports one PICe lane per SATA connector. The edge
adapters can be inserted into a mechanical x1 or greater slot.
IDT Installation of the EB-LOGAN-19 Evaluation Board
Notes
Figure 2.4 PCIe x1 Edge-to-SATA Adapter
Hardware Description
The PES24NT24G2 is a 24-lane, 24-port PCI Express® switch. It is a peripheral chip that performs PCI
Express based switching with a feature set optimized for high performance applications such as servers
and storage. It provides fan-out and switching functions between a PCI Express upstream port and down
stream ports or peer-to-peer switching between downstream ports. Furthermore, up to eight ports can be
configured as NTB ports for multi-root application.
-
The EB-LOGAN-19 Main Board, shown in Figure 2.5, will support up to 4 PCI Express downstream
ports and up to 23 ports when using two 12PACK Daughter Boards.
Basic requirements for the board to run are:
– Host system with a PCI Express root complex supporting x1 configuration through a PCI Express
The PES24NT24G2 requires two differential reference clocks. The EB-LOGAN-19 derives these clocks
from SMA connectors (J17, J20, J66, J67), clock buffer (U50), or SATA connectors (J21, J22) as described
in
IDT Installation of the EB-LOGAN-19 Evaluation Board
Notes
Figure 2.8 Reference Clock Configuration
By default the clock buffer derives its clock from a common source. The common source can be the host
system’s reference clock, the onboard clock generator, or SATA connector (J8). See
The frequency of the global reference clock input may be selected by the Clock Frequency Select
(GCLKFEL) pin to be either 100 MHz or 125 MHz as described in
Global Clock Frequency Switch - SW10[2
SW10[2]Clock Frequency
ON100 MHz (Default)
OFF125 MHz
Table 2.3 Global Reference Input Clock Frequency Select
Table 2.3.
The source for the onboard clock is the ICS841484 clock generator device (U49) connected to a 25MHz
oscillator (X1). When using the onboard clock generator, the output frequency is fixed at 100MHz. There
fore, ICS_FS (S10, bit 1) is ON as the default setting. See Table 2.4.
IDT Installation of the EB-LOGAN-19 Evaluation Board
Notes
Onboard Clock Frequency Switch - S10[1]
S10[1]Clock Frequency
ON100 MHz (Default)
OFF125 MHz
Table 2.4 Onboard Clock Generator Frequency Select
The output of the onboard clock generator is accessible through two yellow colored loop connectors
located on the Evaluation Board. See
capturing purposes and cannot be used to drive the clock from an external source.
Table 2.5. This can be used to connect a scope for probing or
Local Port Input Clocks
Associated with some ports is a port reference clock input (PxCLK). Depending on the port clocking
mode, a differential reference clock is driven into the device on the corresponding PxCLKP and PxCLKN
pins. The frequency of a port reference clock input is always 100 MHz.
Table 2.6 lists the possible sources
for the port reference clock input, and Table 2.7 lists the possible sources for the slot clock input.
Table 2.7 EB-LOGAN-19 Slot Clock Select (Part 1 of 2)
IDT Installation of the EB-LOGAN-19 Evaluation Board
Notes
Slot/Port # HeaderSelection
12J32[1-3 / 2-4] From Clock Buffer (default)
[3-5 / 4-6] SATA (J36)
16J33[1-3 / 2-4] Onboard Clock Generator (U120)
[3-5 / 4-6] From Clock Buffer (default)
[7-9 / 8-10] To P16CLK Clock Header (J15)
[9-11 / 8-10] SATA (J37)
20J34[1-3 / 2-4] From Clock Buffer (default)
[3-5 / 4-6] SATA (J38)
Table 2.7 EB-LOGAN-19 Slot Clock Select (Part 2 of 2)
CLKMODE Selection
All ports in the PES24NT24G2 device (upstream and downstream) use global clocked mode. The port
clocking mode of a port is determined by the state of the CLKMODE[1:0] pins in the boot configuration
vector as shown in
Table 2.8. This field determines the initial value of the Slot Clock Configuration (SCLK)
field in each port’s PCI Express Link Status (PCIELSTS) register. The SCLK field controls the advertisement of whether or not the port uses the same reference clock source as the link partner. A one in the SCLK
field indicates that the port and its link partner use the same reference clock source. This is defined as
Common Clock Configuration by the PCI Express Base Specification. A zero in the SCLK field indicates
that the port and its link partner do not use the same reference clock source.
SW10[8]
CLKMODE[0]
SW10[7]
CLKMODE[1]
Port 0
SCLK
Port[23:1]
SCLK
ONON00
OFFON10
ONOFF01
OFFOFF11
Table 2.8 CLKMODE Selection PES24NT24G2
Power Sources
Power for the PES24NT24G2 and all downstream ports will be generated from the 12V from an external
power connector. See
Table 2.9. A 12V to 3.3V DC-DC converter will be used to provide power to five
switching regulators to generate VDDCORE, VDDPEA, VDDPETA, VDDPEHA, and VDDIO voltages. The
3.3V from the DC-DC converter will be used to power the clock buffers and circuitries.
The external power supply connectors are 24-pin (J69) and 8-pin (J68) molex connector as described in
Tables
2.9 and 2.10. The +12V3 is used to power PES32NT24AG2 and downstream slots 16 and 20. The
IDT Installation of the EB-LOGAN-19 Evaluation Board
Notes
Reset
The PES24NT24G2 supports two types of reset mechanisms as described in the PCI Express specifica-
tion:
– Fundamental Reset: This is a system-generated reset that propagates along the PCI Express
tree through a single side-band signal PERST# which is connected to the Root Complex, the
PES24NT24G2, and the endpoints.
– Hot Reset: This is an In-band Reset, communicated downstream via a link from one device to
another. Hot Reset may be initiated by software. This is further discussed in the PES24NT24G2
User Manual. The EB-LOGAN-19 evaluation board provides seamless support for Hot Reset.
Fundamental Reset
There are two types of Fundamental Resets which may occur on the EB-LOGAN-19 evaluation board:
– Cold Reset: During initial power-on, the onboard voltage monitor (TLC7733D) will assert the PCI
Express Reset (PERSTN) input pin of the PES24NT24G2.
– Warm Reset: This is triggered by hardware while the device is powered on. Warm Reset can be
initiated by two methods:
• Pressing a push-button switch (S3) located on EB-LOGAN-19 board
• The host system board IO Controller Hub asserting PERST# signal, which propagates through
the PCIe upstream edge connector of the EB-LOGAN-19.
Both events cause the onboard voltage monitor (TLC7733D) to assert the PCI Express Reset
(PERSTN) input of the PES24NT24G2 while power is on.
Downstream Reset
Single Partition Mode without Hot Plug:
When the evaluation board initially powers on is assumes the following:
The switch is configured in single partition mode.
Slot 0 is the root port and controls the downstream port resets.
Ports 1-23 are downstream ports.
Hot Plug is disabled.
The following behavior should be observed:
The resets to slots 1-23 should initially be asserted and remain this way until after the fundamental
reset is initially de-asserted.
The assertion of slot 0 reset should propagate to slots 1-23.
Stack Configuration
The PES24NT24G2 contains four stack blocks labeled Stack 0, Stack 1, Stack 2, and Stack 3. Stacks 0
and 1 have four x1 ports each, and stacks 2 and 3 have eight x1 ports each. This provides a total of 24
ports in the device labeled port 0 through port 23.
Table 2.11 lists the ports associated with each stack.
Stacks 0 and 1 have non-mergeable x1 ports. Stacks 2 and 3 may be configured as eight x1 ports, four
x2 ports, two x4 ports, one x8 port, and any combinations in between. The configuration of each stack is
controlled by the Stack Configuration (STK[3:2]CFG) registers. For possible configurations please refer to
the device user manual.
A boot configuration vector consisting of the signals listed in Table 2.12 is sampled by the
PES24NT24G2 during a fundamental reset (while PERSTN is active). The boot configuration vector defines
the essential parameters for switch operation and is set using DIP switches S5, SW8, SW9 and SW10 as
defined in
Table 2.13.
SignalDescription
GCLKFSELGlobal Clock Frequency Select. This pin specifies the frequency of the GCLKP and
GCLKN signals.. Default: low
CLKMODE[1:0]Clock Mode. These pins specify the clocking mode used by switch ports. See Table 2.8 for
a definition of the encoding of these signals. The value of these signals may be overridden
by modifying the Port Clocking Mode (PCLKMODE) register.. Default: 0x3
RSTHALTReset Halt. When this pin is asserted during a switch fundamental reset
sequence, the switch remains in a quasi-reset state with the Master and Slave SMBuses
active. This allows software to read and write registers internal to the device before normal
device operation begins. The device exits the quasi-reset state when the RSTHALT
bit is cleared in the SWCTL register by an SMBus master. Refer to section Switch Fundamental Reset on page 3-2 for further details. Default: low
SSMBADDR[2:1]Slave SMBus Address. SMBus address of the switch on the slave SMBus. Default: 0x3
SWMODE[3:0]Switch Mode. These pins specify the switch operating mode. Default: 0x0
STK2CFG[4:0]Stack 2 Configuration. These pins select the configuration of stack 2 during a switch fun-
STK3CFG[4:0]Stack 3 Configuration. These pins select the configuration of stack 3 during a switch fun-