Artesyn ATCA-9405 Installation

ATCA-9405

Installation and Use

P/N: 6806800M71G
May 2014

©

Copyright 2014 Artesyn Embedded Technologies, Inc.

All rights reserved.

Trademarks

Artesyn Embedded Technologies, Artesyn and the Artesyn Embedded Technologies logo are trademarks and service marks of
Artesyn Embedded Technologies, Inc.© 2014 Artesyn Embedded Technologies, Inc. All other product or service names are the
property of their respective owners.

Intel® is a trademark or registered trademark of Intel Corporation or its subsidiaries in the United States and other countries.
Java™ and all other Java-based marks are trademarks or registered trademarks of Oracle America, Inc. in the U.S. and other countries.
Microsoft®, Windows® and Windows Me® are registered trademarks of Microsoft Corporation; and Windows XP™ is a trademark of

Microsoft Corporation.
PICMG®, CompactPCI®, AdvancedTCA™ and the PICMG, CompactPCI and AdvancedTCA logos are registered trademarks of the PCI

Industrial Computer Manufacturers Group.
UNIX® is a registered trademark of The Open Group in the United States and other countries.

Notice

While reasonable efforts have been made to assure the accuracy of this document, Artesyn assumes no liability resulting from any
omissions in this document, or from the use of the information obtained therein. Artesyn reserves the right to revise this document
and to make changes from time to time in the content hereof without obligation of Artesyn to notify any person of such revision or
changes.

Electronic versions of this material may be read online, downloaded for personal use, or referenced in another document as a URL to
an Artesyn website. The text itself may not be published commercially in print or electronic form, edited, translated, or otherwise
altered without the permission of Artesyn.

It is possible that this publication may contain reference to or information about Artesyn products (machines and programs),
programming, or services that are not available in your country. Such references or information must not be construed to mean that
Artesyn intends to announce such Artesyn products, programming, or services in your country.

Limited and Restricted Rights Legend

If the documentation contained herein is supplied, directly or indirectly, to the U.S. Government, the following notice shall apply
unless otherwise agreed to in writing by Artesyn.

Use, duplication, or disclosure by the Government is subject to restrictions as set forth in subparagraph (b)(3) of the Rights in
Technical Data clause at DFARS 252.227-7013 (Nov. 1995) and of the Rights in Noncommercial Computer Software and
Documentation clause at DFARS 252.227-7014 (Jun. 1995).

Contact Address

Artesyn Embedded Technologies Artesyn Embedded Technologies
Marketing Communications
2900 S. Diablo Way, Suite 190
Tempe, Arizona 85282

Lilienthalstr. 17-19
85579 Neubiberg/Munich
Germany

Regulatory Agency Warnings & Notices

The Artesyn ATCA-9405 meets the requirements set forth by the Federal Communications
Commission (FCC) in Title 47 of the Code of Federal Regulations. The following information is
provided as required by this agency.

This device complies with part 15 of the FCC Rules. Operation is subject to the following two
conditions: (1) This device may not cause harmful interference, and (2) this device must accept
any interference received, including interference that may cause undesired operation.

FCC Rules and Regulations – Part 15

This equipment has been tested and found to comply with the limits for a Class A Class B digital
device, pursuant to part 15 of the FCC Rules. These limits are designed to provide reasonable
protection against harmful interference when the equipment is operated in a commercial
environment in a residential installation. This equipment generates, uses and can radiate radio
frequency energy and, if not installed and used in accordance with the instructions, may cause
harmful interference to radio communications. Operation of this equipment in a residential
area is likely to cause harmful interference, in which case the user will be required to correct the
interference at his own expense.

Making changes or modifications to the ATCA-9405 hardware without the explicit consent of

Artesyn could invalidate the user’s authority to operate this equipment.

EMC Compliance

The electromagnetic compatibility (EMC) tests used an ATCA-9405 model that includes a front
panel assembly from Artesyn.

ATCA-9405 Installation and Use (6806800M71G)

3

Regulatory Agency Warnings & Notices

GR-1089-CORE STANDARD

The intra-building port(s) of the equipment or subassembly is suitable for connection to intra-

building or unexposed wiring or cabling only. The intra-building port(s) of the equipment or
subassembly MUST NOT be metallically connected to interfaces that connect to the OSP or its
wiring. These interfaces are designed for use as intra-building interfaces only (Type 2 or Type 4
ports as described in GR-1089-CORE, Issue 4) and require isolation from the exposed OSP
cabling. The addition of Primary Protectors is not sufficient protection in order to connect
these interfaces metallically to OSP wiring.

4

ATCA-9405 Installation and Use (6806800M71G)

Contents

Contents

Regulatory Agency Warnings & Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

About this Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

1.2 Components and Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

1.3 Functional Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

1.4 Additional Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

1.4.1 Regulatory Compliance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

1.4.2 RoHS Compliance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

1.4.3 Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

1.5 Ordering Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

1.6 Product Identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

2 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

2.2 Electrostatic Discharge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

2.3 ATCA-9405 Circuit Board . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

2.3.1 Switch Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

2.3.1.1 FPGA and CPLD/IPMC Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

2.3.2 Safety Critical Hot Spots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

2.3.3 Connector Pin Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

2.3.3.1 Face Plate Connectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

2.3.3.2 On-board Connectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

2.3.3.3 Back Panel Connectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

2.3.4 Debugging Headers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

2.3.4.1 IPMC Debug Console Header . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

2.3.4.2 COP Header . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

2.3.4.3 EJTAG Header . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

2.4 ATCA-9405 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

2.4.1 Power Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

2.4.2 Environmental Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

2.4.3 Hot Swap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

ATCA-9405 Installation and Use (6806800M71G)

5

Contents

Contents

Contents

2.5 Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

2.5.1 Technical Support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

2.5.2 Product Repair. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

3 Packet Processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

3.2 CN6880 Processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

3.3 Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

3.4 System Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

3.4.1 Memory Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

3.4.2 Memory Socket . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

3.4.3 Memory Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

3.4.4 Thermal Sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

3.5 Octeon U-Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

3.5.1 NVRAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

3.5.2 Network Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

3.6 SerDes Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

3.7 PCI Express Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

3.8 Ethernet Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

3.8.1 Front Panel Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

3.8.2 Base and Fabric Interface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

3.9 Interlaken Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

3.10 USB Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

3.11 UART Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

3.12 I2C Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

3.13 JTAG Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

3.14 Interrupts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

3.14.1 Packet Processor Interrupts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

3.15 Power Supply . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

3.16 Cooling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

4 Service Processor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

4.2 P2020 Processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

6

ATCA-9405 Installation and Use (6806800M71G)

Contents

4.3 Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

4.4 Main Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

4.4.1 Memory Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

4.4.2 Memory Socket . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

4.4.3 Memory Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

4.4.4 Persistent Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

4.4.5 Thermal Sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

4.5 SP U-Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

4.5.1 Environment Variables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

4.5.2 Passing Parameter Set to the Operating System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

4.5.3 Dynamic Variables Set During the Boot Phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

4.5.4 Variables for Controlling the Boot Progress. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

4.5.5 Firmware Update . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

4.5.6 Application/OS Boot. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

4.5.6.1 Default Boot Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

4.5.7 Memory/Address Map Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

4.5.7.1 Address Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

4.6 Local Bus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

4.7 SerDes Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

4.8 PCI Express Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

4.9 Ethernet Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

4.9.1 Front Panel Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

4.9.2 Base and Fabric Interface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

4.10 SPI Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

4.10.1 Boot Flash. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

4.10.2 Boot Flash Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

4.11 USB Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

4.11.1 USB Connector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

4.11.2 e-USB Flash Drive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

4.12 UART Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

4.13 I2C Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

4.13.1 Real Time Clock (RTC) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

4.14 JTAG Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

4.15 Interrupts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

4.15.1 Service Processor Interrupts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

4.16 Cooling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

ATCA-9405 Installation and Use (6806800M71G)

7

Contents

Contents

Contents

5 Ethernet Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

5.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

5.2 Ethernet Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

5.2.1 Port Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

5.2.2 Two-Wire Serial Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94

5.2.3 Switch Management Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94

5.2.4 PHY Management Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

5.3 Base Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

5.4 Fabric Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

5.5 Update Channel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

5.6 Serial Redirection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

6 Service Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .101

6.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

6.2 Port Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .101

6.3 Hot Plug Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

6.3.1 Serial Hot Plug Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

6.4 I2C Slave Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .104

6.5 JTAG Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

6.6 Lane Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .104

7 Mezzanine Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

7.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

8 Intelligent Peripheral Management Controller. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

8.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

8.2 Functional Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

8.3 Firmware Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .114

8.4 HPM.1 Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

8.4.1 FPGA Firmware Upgrade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

8.4.2 Payload Firmware Upgrade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .118

8.4.3 IPMC Firmware Upgrade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

8.4.4 Manual Rollback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

8

ATCA-9405 Installation and Use (6806800M71G)

Contents

8.4.5 Retrieving Versioning Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

8.5 Sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .120

8.5.1 Firmware Progress, OS Boot, and Boot Error Sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128

8.5.2 Boot Bank Supervision Sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128

8.5.3 POST Results Sensor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

8.5.4 Power Good Sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

8.5.5 Power Interface Sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

8.5.6 Reset Cause Sensor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

8.5.7 Presence Sensors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

8.5.8 Voltage and Temperature Sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

8.6 POST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132

8.7 FRU Inventory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133

8.7.1 MAC Address FRU OEM records . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .133

8.8 Reset and Power Domains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

8.9 Power Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

8.10 U-Boot Boot Configuration Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

8.11 Asynchronous Event Notification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

8.12 Serial Line Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

8.13 Built-in Terminal Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

8.13.1 Evaluating the Version of the Telnet Server Firmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142

8.13.2 Establishing a Telnet Session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142

8.14 Fail Safe Logic and Watchdog Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

8.14.1 SP BMC Watchdog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

8.15 Payload Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .145

8.16 Payload Boot Bank Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145

8.17 Settable Graceful Shutdown Timeout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146

8.18 FPGA Health Check . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147

8.19 Local System Event Log (SEL) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147

8.20 IPMI Hardware Watchdog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147

8.21 Artesyn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . OEM Command Set 148

8.21.1 Set Serial Output Command. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148

8.21.2 Get Serial Output Command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150

8.21.3 Set Feature Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .151

8.21.4 Get Feature Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .153

ATCA-9405 Installation and Use (6806800M71G)

9

Contents

Contents

Contents

A Related Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .155

A.1 Artesyn Embedded Technologies - Embedded Computing Documentation . . . . . . . . . . . . . . . 155

Safety Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .157

Sicherheitshinweise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161

10

ATCA-9405 Installation and Use (6806800M71G)

List of Tables

Table 1-1 Regulatory Compliance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Table 1-2 Ordering Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

Table 2-1 Circuit Board Dimensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

Table 2-2 Faceplate Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

Table 2-3 Settings for Switch SW1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

Table 2-4 Settings for Switch SW2 (for debugging only) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

Table 2-5 Settings for Switch SW3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

Table 2-6 Settings for Switch SW4 (for debugging only) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

Table 2-7 USB Connector Pin-out . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

Table 2-8 Ethernet Connector Pin-out . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

Table 2-9 Serial Console Connector Pin-out (Cisco Style) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

Table 2-10 eUSB Drive Connector Pin-out . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

Table 2-11 Zone 1 Connector P10 Pin Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

Table 2-12 Zone 2 Connector P20 Pin Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

Table 2-13 Zone 2 Connector P23 Pin Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

Table 2-14 Zone 3 Connector P30 Pin Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

Table 2-15 Zone 3 Connector P31 Pin Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

Table 2-16 Zone 3 Connector P32 Pin Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

Table 2-17 Debugging Header . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

Table 2-18 IPMC Debug Console Header Pin Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

Table 2-19 COP Header Pin Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

Table 2-20 EJTAG Header Pin Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

Table 2-21 Typical Power Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

Table 2-22 Environmental Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

Table 2-23 Critical Temperature Limits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

Table 3-1 DDR3 Speed Grades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

Table 3-2 CN6880 Memory Module Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

Table 3-3 CN6880 Preferred Memory Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

Table 3-4 CN6880 SerDes Configuration (Interlaken Mode) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

Table 3-5 CN6880 I2C Bus Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

Table 3-6 Packet Processor Interrupts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

Table 4-1 P2020 DDR3 Speed Grades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

Table 4-2 P2020 Memory Module Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

Table 4-3 P2020 Preferred Memory Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

Table 4-4 Dynamic Variables Set During the Boot Phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

Table 4-5 Variables for Controlling the Boot Progress . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

ATCA-9405 Installation and Use (6806800M71G)

11

List of Tables

Table 4-6 Address Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .79

Table 4-7 P2020 SerDes Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

Table 4-8 P2020 eTSEC Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

Table 4-9 P2020 SPI Chip Select Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

Table 4-10 P2020 Boot Flash Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

Table 4-11 P2020 I2C Bus Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

Table 4-12 Service Processor Interrupts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

Table 5-1 Data Plane, Control Plane, and Other Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .89

Table 5-2 Switch Port Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

Table 5-3 Gigabit Ethernet PHY Management Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

Table 5-4 10G Ethernet PHY Management Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .95

Table 6-1 PEX8608 Port Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .102

Table 6-2 Lane Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .104

Table 8-1 HPM.1 Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .116

Table 8-2 ATCA-9405 Specific Sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .120

Table 8-3 Status Sensor's Sensor Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .129

Table 8-4 Voltage and Temperature Sensor Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .130

Table 8-5 FRU Information and SEL at EEPROM Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .133

Table 8-6 Artesyn ECC MAC Address Record . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .133

Table 8-7 Artesyn ECC MAC Address Descriptor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .134

Table 8-8 Interface Type Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .135

Table 8-9 Power Levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .136

Table 8-10 IPMC Boot Parameter Storage Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .138

Table 8-11 TS Channel Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .142

Table 8-12 Artesyn OEM Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .148

Table 8-13 Set Serial Output Command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .148

Table 8-14 Serial Output Selector Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .149

Table 8-15 Get Serial Output Command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .150

Table 8-16 Set Feature Configuration Command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .151

Table 8-17 Feature Selector Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .152

Table 8-18 Get Feature Configuration Command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .153

Table A-1 Artesyn Embedded Technologies - Embedded Computing Publications . . . . . . . . . . . . . .155

12

ATCA-9405 Installation and Use (6806800M71G)

List of Figures

Figure 1-1 General System Block Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Figure 1-2 Declaration of Conformity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

Figure 1-3 Serial Number and Product ID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

Figure 2-1 Faceplate Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

Figure 2-2 Switch Location with Heat Sink SW1 and SW2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

Figure 2-3 Switch Location with Heat Sink SW3 and SW4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

Figure 2-4 Location of the Hot Spots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

Figure 2-5 Location of On-board Connectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Figure 2-6 Location of AdvancedTCA Connectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

Figure 2-7 P10 Backplane Connector Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

Figure 2-8 Air Flow Graph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

Figure 3-1 Packet Processor Unit Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

Figure 4-1 Service Processor Unit Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

Figure 5-1 Ethernet Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

Figure 5-2 Fabric Interface Mode 1x 1000Base-KX (PICMG3.1 Option 1) . . . . . . . . . . . . . . . . . 97

Figure 5-3 Fabric Interface Mode 1x 10GBase-KX4 (PICMG3.1 Option 9) . . . . . . . . . . . . . . . . . 97

Figure 5-4 Fabric Interface Mode 4x 10GBase-KR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

Figure 5-5 Fabric Interface Mode 1x 40GBase-KR4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

Figure 5-6 Update Channel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

Figure 5-7 Serial Redirection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

Figure 6-1 PEX8608 Serial Hot Plug Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

Figure 6-2 PEX8608 Lane Status LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

Figure 7-1 Block Diagram with TCAM Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

Figure 7-2 Block Diagram with PCI-Express Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

Figure 8-1 ATCA-9405 IPMC Block Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

Figure 8-2 Firmware Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

Figure 8-3 FPGA Memory Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

Figure 8-4 IPMC Boot Parameter Storage Configuration Flow . . . . . . . . . . . . . . . . . . . . . . . . . . 137

Figure 8-5 IPMC Serial Line Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140

Figure 8-6 Payload Boot Bank Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146

ATCA-9405 Installation and Use (6806800M71G)

13

List of Figures

14

ATCA-9405 Installation and Use (6806800M71G)

About this Manual

Overview of Contents

This manual is divided into the following chapters and appendix:

 Chapter 1, Introduction, on page 21, provides an overview of the ATCA-9405 board.

 Chapter 2, Setup, on page 31, describes the ATCA-9405 board dimensions, power

requirements, environmental requirements, how to insert and remove the board,
troubleshooting and so on.

 Chapter 3, Packet Processor, on page 59, provides an overview of the CN6880 Processor and

its units.

 Chapter 4, Service Processor, on page 71, provides an overview of the P2020 Processor and

its units.

 Chapter 5, Ethernet Infrastructure, on page 89, describes the ethernet switch, base

interface, fabric interface, update channel interface, and the serial redirection.

 Chapter 6, Service Infrastructure, on page 101, describes the PCI express switch, hot plug

support, I2C slave interface, JTAG support, and lane status.

 Chapter 7, Mezzanine Module, on page 107, provides an overview.

 Chapter 8, Intelligent Peripheral Management Controller, on page 111, provides a functional

overview, firmware architecture, HPM.1 components, Sensors, and so on.

 Appendix A, Related Documentation, on page 155, lists the documents that you can refer.

 Safety Notes on page 157, provides the safety measures to be taken while handling the

board.

 Sicherheitshinweise on page 161, provides the German translation of the Safety Notes.

ATCA-9405 Installation and Use (6806800M71G)

15

Abbreviations

This document uses the following abbreviations:

Abbreviation Definition

ATCA Advanced Telecom Computing Architecture

BIOS Basic Input/Output System

CPLD Complex Programmable Logic Device

DDR Dual Data Rate

DDR3 Double Data Rate 3

DMA Direct Memory Access

DRAM Dynamic Random Access Memory

DXAUI Double Data Rate XAUI

ECC Error Correction Code

About this Manual

About this Manual

EEPROM Electrically Erasable Programmable Read Only Memory

EHCI Enhanced Host Controller Interface

EMC Electro-magnetic Compatibility

ESD Electro-static Discharge

FRU Field Replaceable Unit

GPIO General Purpose Input/Output

I/O Input/Output

2

C Inter Integrated-Circuit Bus (2-wire serial bus and protocol)

I

IMC Integrated Memory Controller

IPMB Intelligent Platform Management Bus

IPMC Intelligent Platform Management Controller

IPMI Intelligent Platform Management Interface

JTAG Joint Test Action Group (test interface for digital logic circuits)

L2 Level 2

LCCB Line Card Clock Building Block

LED Light Emitting Diode

16

ATCA-9405 Installation and Use (6806800M71G)

Abbreviation Definition

LPC Low Pin Count

MAC Medium Access Controller

MMC Module Management Controller

MTBF Mean Time Between Failures

NEBS Network Equipment Building System

NVRAM Non-volatile Random Access Memory

OEM Original Equipment Manufacturer

OOS Out-of-service

PCB Printed Circuit Board

PCIe PCI-Express

PHY Physical layer device (for Ethernet)

About this Manual

PICMG PCI Industrial Computer Manufacturers Group

PLL Phase Locked Loop

POST Power-on Self Test

RoHS Restriction of Hazardous Substances

RTC Real-Time Clock

RTM Rear Transition Module

Rx Receive line (of a duplex serial communication interface)

RXAUI Reduced XAUI

SDR Sensor Data Record

SDRAM Synchronous Dynamic Random Access Memory

SECDED Single Error Correction Double Error Detection

SELV Safety Extra Low Voltage

SerDes Serializer/Deserializer

SGMII Serial Gigabit Media Independent Interface

SMP Symmetric Multi-Processors

SPD Serial Presence Detect

TCAM Ternary Content Addressable Memory

ATCA-9405 Installation and Use (6806800M71G)

17

Abbreviation Definition

TCP Transmission Control Protocol

Tx Transmit line (of a duplex serial communication interface)

UART Universal Asynchronous Receiver-Transmitter

USB Universal Serial Bus

UTMI USB 2.0 Transceiver Macrocell Interface

XAUI 10 Gigabit Attachment Unit Interface

XLAUI 40 Gigabit Attachment Unit Interface

Conventions

The following table describes the conventions used throughout this manual.

About this Manual

About this Manual

Notation Description

0x00000000 Typical notation for hexadecimal numbers (digits are

0 through F), for example used for addresses and
offsets

0b0000 Same for binary numbers (digits are 0 and 1)

bold Used to emphasize a word
Screen Used for on-screen output and code related elements

or commands in body text

Courier + Bold Used to characterize user input and to separate it

from system output

Reference Used for references and for table and figure

descriptions

File > Exit Notation for selecting a submenu

<text> Notation for variables and keys

[text] Notation for software buttons to click on the screen

and parameter description

... Repeated item for example node 1, node 2, ..., node

12

18

ATCA-9405 Installation and Use (6806800M71G)

Notation Description

About this Manual

.
.
.

.. Ranges, for example: 0..4 means one of the integers

| Logical OR

Omission of information from example/command
that is not necessary at the time being

0,1,2,3, and 4 (used in registers)

Indicates a hazardous situation which, if not avoided,
could result in death or serious injury

Indicates a hazardous situation which, if not avoided,
may result in minor or moderate injury

Indicates a property damage message

No danger encountered. Pay attention to important
information

Summary of Changes

This manual has been revised and replaces all prior editions.

Part Number Publication Date Description

6806800M71A September 2011 Preliminary copy

6806800M71B November 2011 DA version

6806800M71C June, 2012 EA version

ATCA-9405 Installation and Use (6806800M71G)

19

About this Manual

About this Manual

Part Number Publication Date Description

6806800M71D December, 2012 GA version

6806800M71E May, 2013 Updated Table 2-21, Establishing a Telnet Session on page

142, and IPMI command examples in Set Serial Output

Command on page 148.

6806800M71F November, 2013 Updated Chapter 8, Establishing a Telnet Session, on page

142, Figure 1-1 on page 23, Table 4-12 on page 87, Figure
5-1, Table 5-2 on page 92, Table 5-4 on page 95, Chapter 5,
Fabric Interface, on page 96 and Figure 7-1.

6806800M71G May, 2014 Re- branded to Artesyn template.

20

ATCA-9405 Installation and Use (6806800M71G)

Introduction

1.1 Overview

The ATCA-9405 is an Advanced Telecom Computing Architecture (AdvancedTCA®, ATCA®)
blade based on two Cavium OCTEON II CN6880 Multi-Core MIPS64 Packet Processors and the
dual-core Freescale P2020 Service Processor.

The MIPS64 processor cores can be configured for up to 32-way SMP LINUX support, run
Cavium Simple Executives, and run fast path packet processing software for time critical
applications. The ATCA-9405 supports for up to 162 Gigabit Ethernet (GbE) I/O bandwidth and
a redundant 40GbE fabric. It allows the design of compact application systems for the
upcoming new IP data infrastructure.The ATCA-9405 represents a performance and
throughput increase over previous generations.

The dual-core Free scale QorIQ™ P2020 processor is used for basic board setup, general board
management, and Ethernet switch management. The management processor has local mass
storage support for uboot and user code. The boot code for the OCTEON II packet processors
is provided via the management processor. The dual-core processor is used to offload other
blade functions in order to maximize the packet processing capability, including managing
Layer 2 and 3 switching/routing functions on the local Ethernet switch.

Chapter 1

This blade is targeted at security and packet-processing applications in the wireless and
transport market segments. These markets include dataplane packet-processor, security co­processor, video compression, and pattern matching.

The ATCA-9405 complies with the SCOPE recommended profile for central office ATCA
systems, PICMG® 3.0 ATCA mechanical specifications, E-keying, and Hot Swap.

ATCA-9405 Installation and Use (6806800M71G)

21

1.2 Components and Features

The ATCA-9405 hardware components and its features are:

 Single-slot ATCA form factor

 Two Cavium OCTEON II CN6880 Multi-Core MIPS64 Packet Processors each with memory

support up to 64 GB DDR3

 Dual Core Free scale P2020 Service Processor for blade management with up to 8 GB DDR3

Memory

 Marvell 98CX8234 32-port Ethernet switch connecting all rear I/O, backplane I/O and

Packet Processors with L2 and L3 switch management software.

 Dual 1G/10G/40G Ethernet fabric interface

 Dual 1G Ethernet base interface

 Support for Wind River PNE 4.x OS, Cavium SDK, and 6WIND 6WINDGate

Introduction

22

ATCA-9405 Installation and Use (6806800M71G)

1.3 Functional Overview

The following block diagram provides a functional overview of the ATCA-9405.

Figure 1-1 General System Block Diagram

Introduction

ATCA-9405 Installation and Use (6806800M71G)

23

1.4 Additional Information

This section lists the regulatory certifications of ATCA-9405 hardware and briefly discusses the
terminology and notation conventions used in this manual.

Mean time between failures (MTBF) has been calculated at 439,924 hours using the Telcordia
SR-332, Issue 1 (Reliability Prediction for Electronic Equipment), method 2 at 30°C.

1.4.1 Regulatory Compliance

The ATCA-9405 has been tested to comply with various standards:

Table 1-1 Regulatory Compliance

Item Description

Introduction

Designed to comply with NEBS,
Level 3

Designed to comply with ETSI ETSI Storage, EN 300 019-1-1, Class 1.2 equipment, Not

Telcordia GR-63-CORE, NEBS Physical Protection

Telcordia GR-1089-CORE, Electromagnetic Compatibility and
Electrical Safety – Generic Criteria for Network
Telecommunications Equipment. Equipment Type 2

Temperature Controlled Storage Locations

ETSI Transportation, EN 300 019-1-2, Class 2.3 equipment,
Public Transportation

ETSI Operation, EN 300 019-1-3, Class 3.1(E) equipment,
Temperature Controlled Locations

ETSI EN 300 132-2 Environmental Engineering (EE); Power
supply interface at the input to telecommunications
equipment; Part 2: Operated by direct current (dc)

ETSI ETS 300 753, Equipment Engineering (EE); Acoustic noise
emitted by telecommunications equipment

24

ATCA-9405 Installation and Use (6806800M71G)

Introduction

Table 1-1 Regulatory Compliance (continued)

Item Description

EMC ETSI EN 300 386 Electromagnetic compatibility and Radio

spectrum Matters (ERM); telecommunication network
equipment; ElectroMagnetic Compatibility (EMC)
requirements, Telecommunication equipment room
(attended)

CFR 47 FCC Part 15 Subpart B, Class A (US); FCC Part 15 - Radio
Frequency Devices; Subpart B: Unintentional Radiators

AS/NZS CISPR 22 (Australia/New Zealand), Limits and Methods
of Measurement of Radio Disturbance Characteristics of
Information Technology Equipment

VCCI Class A (Japan), Voluntary Control Council for Interference
by Information Technology Equipment

CISPR 22 Information technology equipment – Radio
disturbance characteristics – Limits and methods of
measurement

Safety Certified to UL/CSA 60950-1, EN 60950-1 and IEC 60950-1 CB

RoHS/WEEE compliance DIRECTIVE 2002/95/EC OF THE EUROPEAN PARLIAMENT AND

Interoperability Designed to operate within a CP-TA B.4 system environment at

ATCA-9405 Installation and Use (6806800M71G)

CISPR 24 Information technology equipment – Immunity
characteristics – Limits and methods of measurement

Scheme

Safety of information technology equipment, including
electrical business equipment

OF THE COUNCIL on the restriction of the use of certain
hazardous substances in electrical and electronic equipment
(RoHS)

DIRECTIVE 2002/96/EC OF THE EUROPEAN PARLIAMENT AND
OF THE COUNCIL on waste electrical and electronic equipment
(WEEE)

full performance

25

1.4.2 RoHS Compliance

The ATCA-9405 is compliant with the European Union’s RoHS (Restriction of use of Hazardous
Substances) directive created to limit harm to the environment and human health by
restricting the use of harmful substances in electrical and electronic equipment. Effective July
1, 2006, RoHS restricts the use of six substances: cadmium (Cd), mercury (Hg), hexavalent
chromium (Cr (VI)), polybrominated biphenyls (PBBs), polybrominated diphenyl ethers
(PBDEs), and lead (Pb). Configurations that are RoHS compliant are built with lead-free solder.

Introduction

26

ATCA-9405 Installation and Use (6806800M71G)

Figure 1-2 Declaration of Conformity

Introduction

ATCA-9405 Installation and Use (6806800M71G)

27

1.4.3 Notation

Active low signals

An active low signal is indicated with an asterisk # after the signal name.

1.5 Ordering Information

Table 1-2 Ordering Information

Part Number Product Description

ATCA-9405B-32GB ATCA-9405 - 2X CN6880-1.2GHZ, 2X 16GB DDR3 MEMORY ON FOUR

LOCAL MEMORY CONTROLLERS - 1X P2020 WITH 2GB MEMORY - 16GB
FLASH - BBS - SWITCH MGMT SW

Introduction

ATCA-9405B-64GB On customer request

ARTM-9405B-16X10GE ARTM-9405 - 8X10G (SFP+) and 2x40G (QSFP)

SM-BBS-WR-ATCA-9405 Basic Blade Services software and L2 & L3 switch management software

based on Wind River PNE4.x Linux. (CD Media only)

RJ45-DSUB-ATCA7140 RJ-45 DSUB cable for the ATCA-7140, 7150, 7350, 736X

28

ATCA-9405 Installation and Use (6806800M71G)

1.6 Product Identification

Figure 1-3 Serial Number and Product ID

Introduction

On R1.0 prototypes, only the Serial Number label is available.

ATCA-9405 Installation and Use (6806800M71G)

29

Introduction

30

ATCA-9405 Installation and Use (6806800M71G)

Setup

2.1 Overview

This chapter describes the physical layout of the board, the setup process, and how to check for
proper operation once the board has been installed. This chapter also includes
troubleshooting, service, and warranty information.

2.2 Electrostatic Discharge

Before you begin the setup process, remember that the electrostatic discharge (ESD) can easily
damage the components on the ATCA-9405 hardware. Electronic devices, especially those
with programmable parts, are susceptible to ESD, which can result in operational failure.
Unless you ground yourself properly, static charges can accumulate in your body and cause
ESD damage when you touch the board.

Chapter 2

Use proper static protection and handle ATCA-9405 boards only when absolutely necessary.
Always wear a wriststrap to ground your body before touching a board. Keep your body
grounded while handling the board. Hold the board by its edges—do not touch any
components or circuits. When the board is not in an enclosure, store it in a static-shielding
bag.

To ground yourself, wear a grounding wriststrap. Simply placing the board on top of a static­shielding bag does not provide any protection—place it on a grounded dissipative mat. Do not
place the board on metal or other conductive surfaces.

ATCA-9405 Installation and Use (6806800M71G)

31

2.3 ATCA-9405 Circuit Board

The ATCA-9405 circuit board is an ATCA blade assembly and complies with the PICMG 3.0 ATCA
mechanical specification. It uses a 16-layer printed circuit board with the following dimensions:

Table 2-1 Circuit Board Dimensions

Width Depth Height Weight (typical)

Setup

12.687 in.
(322.25 mm)

This is the typical weight for the ATCA-9405. Board weight varies slightly per configuration; contact
Technical Support if you require a specific configuration weight.

11.024 in.
(280.01 mm)

< .84 in.
(<21.33 mm)

8.38 lb. (3.80 kg)

The Figure 2-1 on page 32, shows the face plate layout for ATCA-9405 board.

Figure 2-1 Faceplate Layout

The face plate of the blade provides the following interfaces and control elements.

Table 2-2 Faceplate Interfaces

Processor Interfaces / Control Elements

Packet Processor One Serial Console port for Packet Processor 1 (only for maintenance

purposes)
One Serial Console port for Packet Processor 2 (only for maintenance

purposes)
One 10/100/1000Base-T Ethernet port shared for both Packet

Processors (only for maintenance purposes)

32

Service Processor One USB port (only for maintenance purposes)

One Serial Console port (only for maintenance purposes)
One 10/100/1000Base-T Ethernet port (only for maintenance purposes)

ATCA-9405 Installation and Use (6806800M71G)

Table 2-2 Faceplate Interfaces (continued)

Processor Interfaces / Control Elements

IPMC Out of Service (OOS) LED

In Service (IS) LED
Attention (ATN) LED
Hot Swap (H/S) LED
Recessed Reset Button

For these interfaces, the maximum cable length should not exceed more than 3m.

Setup

2.3.1 Switch Settings

All mechanical switches are OFF in their default configuration. Switch OFF means high level for
the connected signal and switch ON means low level for the connected signal.

ATCA-9405 Installation and Use (6806800M71G)

33

Setup

The switches are placed on component side 1. The Figure 2-2 on page 34, shows the switch
locations on the board.

Figure 2-2 Switch Location with Heat Sink SW1 and SW2

34

ATCA-9405 Installation and Use (6806800M71G)

Figure 2-3 Switch Location with Heat Sink SW3 and SW4

Setup

2.3.1.1 FPGA and CPLD/IPMC Switches

The switch settings are described in Table 2-3 on page 35.

Table 2-3 Settings for Switch SW1

Switch Description Default

SW1-1 IPMI Boot Select

OFF = SPI Boot Flash is selected by IPMC
ON = SPI Boot Flash is selected by SW1-2

SW1-2 Manual Boot Select

OFF = Default SPI Boot Flash selected
ON = Recovery SPI Boot Flash selected

ATCA-9405 Installation and Use (6806800M71G)

OFF Default or Recovery SPI Boot Flash

is selected by IPMC

OFF Default SPI Boot Flash is used

35

Table 2-3 Settings for Switch SW1 (continued)

Switch Description Default

SW1-3 Debug Header Boot Select

This switch overrides selection done via
SW1-1 and SW1-2.

OFF = Boot from SPI Boot Flash (TSOP
devices)

ON = Boot from SPI Debug Header

SW1-4 Reserved OFF Do not change

OFF Boot from TSOP SPI Flash

Table 2-4 Settings for Switch SW2 (for debugging only)

Switch Description Default

SW2-1 Reserved OFF Do not change

SW2-2 Reserved OFF Do not change

Setup

SW2-3 Reserved OFF Do not change

SW2-4 Reserved OFF Do not change

Table 2-5 Settings for Switch SW3

Switch Description Default

SW3-1 Enable E10-USB Emulator for H8S

OFF = Disabled
ON = Enabled

SW3-2 Enable H8S Programming via Debug Console

OFF = Disabled
ON = Enabled

SW3-3 Manual Payload Power Enable for Blade

OFF = IPMI controlled power enable for blade
ON = Manual power enable for blade

SW3-4 Manual Payload Power Enable for RTM

OFF = IPMI controlled power enable for RTM
ON = Manual power enable for RTM

OFF E10-USB Emulator disabled

OFF H8S programming disabled

OFF IPMI controlled power enabled

OFF IPMI controlled power enabled

36

ATCA-9405 Installation and Use (6806800M71G)

Table 2-6 Settings for Switch SW4 (for debugging only)

Switch Description Default

Setup

SW4-1 Manual Power Enable for Packet Processor 1

OFF = SP controlled PP1 power enable
ON = Manual PP1 power enable

SW4-2 Manual Power Enable for Packet Processor 2

OFF = SP controlled PP2 power enable
ON = Manual PP2 power enable

SW4-3
(dual

function)

SW4-4 IPMC Console Output Selection

IPMC Watchdog Enable (SW3-3 = OFF)
OFF = Watchdog enabled
ON = Watchdog disabled

Reserved (SW3-3 = ON) OFF Do not change

OFF = IPMC console output via debug
connector

ON = IPMC console output via front panel
connector

OFF Software controlled (SP) power

enable for PP1

OFF Software controlled (SP) power

enable for PP2

OFF IPMC Watchdog Enabled

OFF IPMC Console Output via

debugging header

ATCA-9405 Installation and Use (6806800M71G)

37

2.3.2 Safety Critical Hot Spots

The following figure shows the critical hot spots on the ATCA-9405 board.

Figure 2-4 Location of the Hot Spots

Setup

38

Temperature Spot 2 (on 48V DC/DC
Converter) = 100°C (max)
exact location: in the geometric middle
of the heat spreader

Temperature Spot 1 (on Power Entry

Module) = 110°C (max)
exact location: on top of upper transformer
housing

ATCA-9405 Installation and Use (6806800M71G)

2.3.3 Connector Pin Assignment

2.3.3.1 Face Plate Connectors

Refer Figure 2-1 for the face plate layout.

2.3.3.1.1 USB Port

One standard USB Type-A connector for access to the USB interface of the Service Processor is
available at the face plate.

Table 2-7 USB Connector Pin-out

Pin Signal

1 VCC_USB

2D-

Setup

3D+

4GND

2.3.3.1.2 Ethernet Ports

Two shielded RJ45 connectors with integrated transformers for 10/100/1000Base-T Ethernet
are available at the front panel.

Table 2-8 Ethernet Connector Pin-out

Pin Signal

1 BI_DA+

2 BI_DA-

3 BI_DB+

4 BI_DC+

5 BI_DC-

6 BI_DB-

7 BI_DD+

8 BI_DD-

ATCA-9405 Installation and Use (6806800M71G)

39

2.3.3.1.3 Serial Console Ports

Three shielded RJ45 connectors (with Cisco style pin-out) for serial consoles are available at the
front panel.

Table 2-9 Serial Console Connector Pin-out (Cisco Style)

Pin Signal

1RTS#

2NC

3 TXD

4GND

5GND

6 RXD

7NC

Setup

8CTS#

40

ATCA-9405 Installation and Use (6806800M71G)

2.3.3.2 On-board Connectors

The following figure shows the location of eUSB Drive Connector (P6) on ATCA-9405.

Figure 2-5 Location of On-board Connectors

Setup

ATCA-9405 Installation and Use (6806800M71G)

P6

41

2.3.3.2.1 eUSB Drive Connector

ATCA-9405 provides one 2x5 pin header and a mounting hole for eUSB drives like Smart
Modular Z-U130.

Table 2-10 eUSB Drive Connector Pin-out

Pin Signal

1VCC

2NC

3D-

4NC

5D+

6NC

7GND

Setup

8NC

9KEY

10 NC

42

ATCA-9405 Installation and Use (6806800M71G)

2.3.3.3 Back Panel Connectors

2.3.3.3.1 Overview

The AdvancedTCA backplane connectors reside in three zones, 1 to 3 as specified by the
AdvancedTCA standard and are called P10, P20 and 23, and P30, P31, and 32. The pinouts of
all these connectors are given in this section.

Figure 2-6 Location of AdvancedTCA Connectors

Setup

ATCA-9405 Installation and Use (6806800M71G)

43

2.3.3.3.2 Zone 1 Connector

The Zone 1 connector is used to deliver power from the ATCA back plane to the ATCA-9405
baseboard. The pin out for this connector is defined by the ATCA specifications. The
ATCA-9405 does not implement Metallic Test or Ringing Generator Circuitry and therefore
does not use pins 17 through 24.

Figure 2-7 P10 Backplane Connector Pinout

Setup

44

Table 2-11 Zone 1 Connector P10 Pin Assignment

Contact Destination Description

1 - 4 Reserved Reserved

5 IPMC ISC PC0 Hardware Address Bit 0

6 IPMC ISC PC1 Hardware Address Bit 1

7 IPMC ISC PC2 Hardware Address Bit 2

8 IPMC ISC PC3 Hardware Address Bit 3

9 IPMC ISC PD4 Hardware Address Bit 4

10 IPMC ISC PD5 Hardware Address Bit 5

11 IPMC ISC PD6 Hardware Address Bit 6

12 IPMC ISC PD7 Hardware Address Bit 7

ATCA-9405 Installation and Use (6806800M71G)

Table 2-11 Zone 1 Connector P10 Pin Assignment (continued)

Contact Destination Description

13 IPMC IMC PD0 IPMB Clock Port A

14 IPMC IMC PD1 IPMB Data Port A

15 IPMC ISC PC5 IPMB Clock Port B

16 IPMC ISC PC4 IPMB Data Port A

17-24 Not used Not used

25 Shelf Ground Shelf Ground

26 Logic Ground Logic Ground

27 Power Building Block Enable B

28 Power Building Block Voltage Return A

29 Power Building Block Voltage Return B

30 Power Building Block Early -48V A

Setup

31 Power Building Block Early -48V B

32 Power Building Block Enable A

33 Power Building Block -48V A

34 Power Building Block -48V A

2.3.3.3.3 Zone 2 Connector

The ATCA specifications define five identical connectors for the Zone 2 data transport. These
connectors are referred to as Free Board connectors and are assigned reference designators
P20 through P24 as per the ATCA specifications.

The ATCA-9405 is a node board and uses only two of the five connectors P20 and P23. P20 is
used to support clock synchronization interface and update channels and P23 is used to
support the base and fabric channels.

ATCA-9405 Installation and Use (6806800M71G)

45

Setup

Tyco HM-Zd Plus connectors are used to support 40 Gigabit Ethernet through backplane.

Table 2-12 Zone 2 Connector P20 Pin Assignment

Row Interface a b c d e f g h

1 CLK CLK1A+ CLK1A- CLK1B+ CLK1B - CLK2A+ CLK2A- CLK2B+ CLK2B-

2 Update

Channel

3 UC_TX2+ UC_TX2- UC_RX2+ UC_RX2- UC_TX3+ UC_TX3- UC_RX3+ UC_RX3-

4 UC_TX0+ UC_TX0- UC_RX0+ UC_RX0- UC_TX1+ UC_TX1- UC_RX1+ UC_RX1+

5Fabric

Channel 15

6NCNCNCNCNCNCNCNC

7Fabric

Channel 14

8NCNCNCNCNCNCNCNC

9Fabric

Channel 13

10 NC NC NC NC NC NC NC NC

NC NC NC NC CLK3A+ CLK3A- CLK3B+ CLK3B-

NC NC NC NC NC NC NC NC

NC NC NC NC NC NC NC NC

NC NC NC NC NC NC NC NC

46

Table 2-13 Zone 2 Connector P23 Pin Assignment

Row Interface a b c d e f g h

1Fabric

Channel 2

2 FC2_TX0+ FC2_TX0- FC2_RX0+ FC2_RX0-FC2_TX1+FC2_TX1- FC2_RX1+ FC2_RX1-

3Fabric

Channel 1

4 FC1_TX0+ FC1_TX0- FC1_RX0+ FC1_RX0-FC1_TX1+FC1_TX1- FC1_RX1+ FC1_RX1-

5Base

Channel 1

6Base

Channel 2

7NA NC NC NC NC NC NC NC NC

8NA NC NC NC NC NC NC NC NC

9NA NC NC NC NC NC NC NC NC

10 NA NC NC NC NC NC NC NC NC

FC2_TX2+ FC2_TX2- FC2_RX2+ FC2_RX2-FC2_TX3+FC2_TX3- FC2_RX3+ FC2_RX3-

FC1_TX2+ FC1_TX2- FC1_RX2+ FC1_RX2-FC1_TX3+FC1_TX3- FC1_RX3+ FC1_RX3-

BC1_DA+ BC1_DA- BC1_DB+ BC1_DB - BC1_DC+BC1_DC- BC1_DD+ BC1_DD-

BC2_DA+ BC2_DA- BC2_DB+ BC2_DB - BC2_DC+BC2_DC- BC2_DD+ BC2_DD-

ATCA-9405 Installation and Use (6806800M71G)

2.3.3.3.4 Zone 3 Connector

ATCA-9405 provides three connectors P30, P31, and P32 for connection to the RTM.

Table 2-14 Zone 3 Connector P30 Pin Assignments

Setup

Table 2-15 Zone 3 Connector P31 Pin Assignments

ATCA-9405 Installation and Use (6806800M71G)

47

Table 2-16 Zone 3 Connector P32 Pin Assignments

2.3.4 Debugging Headers

Setup

The following debugging headers are provided on the ATCA-9405. Please note that debugging
headers are only used internally and may be removed later for cost reduction reasons. For more
information regarding location and usage of debugging headers, please contact
sales/marketing team.

Table 2-17 Debugging Header

Location Type Description

JP1 16-pin Header, 1.27mm pitch P2020 debugging header

MSRCID signals and second serial console port (UART1
interface)

JP2 8-pin Header, 1.27mm pitch P2020 Boot Flash header

JP3 3-pin Header, 2.54mm pitch IPMC debug console

JP5 16-pin Header, 1.27mm pitch E10A-USB Emulator header for IPMC

P47 16-pin Header, 1.27mm pitch Packet Processor 1 debugging header

EJTAG support and second serial console port (UART1
interface)

P57 16-pin Header, 1.27mm pitch Packet Processor 2 debugging header

EJTAG support and second serial console port (UART1
interface)

48

ATCA-9405 Installation and Use (6806800M71G)

Table 2-17 Debugging Header (continued)

Location Type Description

P7 16-pin Header, 1.27mm pitch P2020 COP interface

P8 16-pin Header, 1.27mm pitch Management and Payload Power JTAG interface

2.3.4.1 IPMC Debug Console Header

A three pin header is provided to access IPMC debug console. The pin assignment is shown
below.

Table 2-18 IPMC Debug Console Header Pin Assignment

Pin Signal

1GND

3 IPMC_TXD

Setup

4 IPMC_RXD

2.3.4.2 COP Header

A 16-pin header is provided for access to P2020 COP interface. The pin assignment is shown
below.

Table 2-19 COP Header Pin Assignment

Pin Signal

1 TDO

2QACK

3 TDI

4TRST#

5RUNSTOP

6 Target Power

7TCK

8 CKSTP_IN#

ATCA-9405 Installation and Use (6806800M71G)

49

Table 2-19 COP Header Pin Assignment (continued)

Pin Signal

9TMS

10 NC

11 SRESET#

12 GND

13 HRESET#

14 KEY

15 CKSTP_OUT#

16 GND

2.3.4.3 EJTAG Header

Setup

A separate 16-pin header is provided to access each Packet Processor EJTAG interface. The pin
assignment is shown below.

Table 2-20 EJTAG Header Pin Assignment

Pin Signal

1TRST#

2 UART1_RXD

3 TDI

4 UART1_TXD

5 TDO

6GND

7TMS

8GND

9TCK

10 NC

11 RESET#

12 KEY

50

ATCA-9405 Installation and Use (6806800M71G)

Table 2-20 EJTAG Header Pin Assignment (continued)

Pin Signal

13 NC

14 Target Power

15 NC

16 NC

The standard EJTAG connector has only 14 pins, thus do not connect to pin 15 and 16.

Setup

2.4 ATCA-9405 Setup

Following items are required to setup and check the operation of the ATCA-9405:

 ATCA chassis and power supply

 Console cables for EIA-232 ports (Cisco style pin out)

 Computer terminal

Save the antistatic bag and box for future shipping or storage.

ATCA-9405 Installation and Use (6806800M71G)

51

2.4.1 Power Requirements

Make sure that the blade is used in an ATCA shelf connected to -60 VDC up to -48 VDC,
according to Telecommunication Network Voltage (TNV-2). A TNV-2 circuit is a circuit whose
normal operating voltages exceed the limits for a safety-extra-low-voltage (SELV) under
normal operating conditions, and which is not subject to over-voltages from
telecommunication networks.

Table 2-21 Typical Power Requirements

Configuration Power

Dual CN6880 running at 1.2GHz with 16GB DDR3 memory each,
P2020 running at 1.0GHz with 2GB DDR3 memory, 16GB USB Flash
Drive, Linux booted on P2020, Ethernet Switch fully initialized,
Cavium Octeon RiscCore Stress test running on all 64 CN6880 cores,
board running at room temperature.

Dual CN6880 running at 1.0 GHz with 16GB DDR3 memory each,
P2020 running at 1.0GHz with 2GB DDR3 memory, 16GB USB Flash
Drive, Linux booted on P2020, Ethernet Switch fully initialized,
Cavium Octeon RiscCore Stress test running on all 64 CN6880 cores,
board running at room temperature.

Dual CN6880 running at 800 MHz with 16GB DDR3 memory each,
P2020 running at 1.0GHz with 2GB DDR3 memory, 16GB USB Flash
Drive, Linux booted on P2020, Ethernet Switch fully initialized,
Cavium Octeon RiscCore Stress test running on all 64 CN6880 cores,
board running at room temperature.

Setup

235 W

215 W

195 W

52

ARTM-9405B-16X10GE (maximum Power consumption) 36 W

Rated Voltage
Exception in the US and Canada

Operating Voltage
Exception in the US and Canada

-48 VDC to -60 VDC

-48 VDC

-39 VDC to -72 VDC

-39 VDC to -60 VDC

The exact power requirements for the ATCA-9405 circuit board depends upon the specific
configuration of the board, including the CPU frequency and amount of memory installed on
the board.

ATCA-9405 Installation and Use (6806800M71G)

2.4.2 Environmental Considerations

As with any printed circuit board, make sure that the air flow to the board is adequate. Chassis
constraints and other factors greatly affect the air flow rate. The environmental requirements
are as follows:

Table 2-22 Environmental Requirements

Requirement Operating Non-Operating

Setup

Temperature +5 ºC (+41 °F) to +40 ºC (+104 °F)

(normal operation) according to NEBS
Standard GR-63-CORE

-5 ºC (+23 °F) to +55 ºC (+131 °F)
(exceptional operation) according to
NEBS Standard GR-63-CORE

Airflow The blade is designed to operate in a

chassis that provides 35 CFM across the
blade for the stated temperature range

Temperature
change

Relative humidity 5% to 90% non-condensing according to

Vibration 1 g from 5 to 100 Hz and back to 5 Hz at

Shock Half-sine, 11 m/Sec, 30 mSec/sec2 Blade level packaging

+/- 0.5 ºC/min according to NEBS
Standard GR-63-CORE

Artesyn Embedded Technologies­internal environmental

requirements

a rate of 0.1 octave/minute

-40 ºC (-40 °F) to +70 ºC (+158 °F)
(may be further limited by installed
accessories)

-

+/- 0.5 ºC/min

5% to 95% non-condensing
according to Artesyn Embedded

Technologies-internal
environmental requirements

5-20 Hz at 0.1 g2/Hz
20-200 Hz at -3.0 dB/octave
Random 20-200 Hz at -3 m/Sec2

Half-sine, 6 mSec at 180 m/Sec2

Free fall 1,200 mm/all edges and corners

ATCA-9405 Installation and Use (6806800M71G)

1.0 m (packaged)
100 mm (unpacked)

53

Figure 2-8 Air Flow Graph

Setup

During the safety qualification of this blade, the following on-board locations were identified
as critical with regards to the maximum temperature during blade operation. To guarantee
proper blade operation and to ensure safety, you have to make sure that the temperatures at
the locations specified in the following table are not exceeded. If not stated otherwise, the
temperatures should be measured by placing a sensor exactly at the given locations.

Table 2-23 Critical Temperature Limits

Maximum Case or Junction

Component Thermal Design Power

Cavium CN6880 PP1 72 W Tj = 101°C

Cavium CN6880 PP2 72 W Tj = 101°C

DDR3 DIMM Modules 6.9 W Tc = 85°C

PCI-Express Switch 2.9 W Tj = 110°C

Ethernet Switch 33.5 W Tj = 115°C

Temperature

If you integrate the blade in your own system, contact your local sales representative for
further safety information.

54

ATCA-9405 Installation and Use (6806800M71G)

2.4.3 Hot Swap

The ATCA-9405 can be Hot Swapped, as defined in the ATCA specification. This section
describes how to insert and extract an ATCA-9405 module in a typical ATCA system. (These
procedures assume the system is using a shelf manager.)

The ATCA-9405 Rear Transition Module (RTM) has its own Hot Swap LED and switch, and it
can be Hot Swapped in/out independently of the face plate. If the face plate is not present,
then the RTM will not be powered. If the face plate is Hot Swapped out, the blue LED of the
RTM illuminates. In either case, the RTM can be safely removed.

Personal Injury or Product Damage

Setup

The product is supplied by a TNV-2 voltage. This voltage is considered hazardous. Make sure
that the external power supply meets the relevant safety standards.

Make sure that TNV-2 is separated from dangerous voltages (mains) through double or
reinforced insulation.

Inserting a board

1. Insert the ATCA-9405 into an available slot.

2. Push the face plate handle (tab).

The blue Hot Swap LED on the front panel (see Figure 2-1) flashes a long blink to indicate that
the board insertion is in progress and the system management software is activating the slot.
Then the blue LED turns off, indicating the insertion process is complete, and payload power is
present.

ATCA-9405 Installation and Use (6806800M71G)

55

Removing a board

1. Pull out the handle (tab) on the ATCA-9405 front panel at one click.
A short blink indicates that the board is requesting permission for extraction.

2. Remove the board when the blue LED on the front panel is on (no payload power).

Do not remove the ATCA-9405 while the blue LED is blinking.

2.5 Troubleshooting

In case of difficulty, use the following checklist:

Setup

 Check that the ATCA-9405 circuit board is seated firmly in the carrier

 Check that the system is not overheating

 Check the cables and connectors to be certain that they are secure

 Check that your terminal is connected to a console port

2.5.1 Technical Support

If you need help in resolving a problem with your ATCA-9405, visit

www.artesyn.com/computing. Keep the following information ready:

 ATCA-9405 serial number identification, see Figure "Serial Number and Product ID" on

page 29.

 Version and part number of the operating system (if applicable)

 Whether your board has been customized for options such as a higher processor speed or

additional memory

 License agreements (if applicable)

56

ATCA-9405 Installation and Use (6806800M71G)

2.5.2 Product Repair

If you plan to return the board to Artesyn Embedded Technologies for service, visit

www.artesyn.com/computing to obtain a Return Merchandise Authorization (RMA) number.

List the items that you are returning and the board serial number, plus your purchase order
number and billing information if your ATCA-9405 hardware is out of warranty. Contact our
Test and Repair Services Department for any warranty questions. If you return the board, be
sure to enclose it in an antistatic bag, such as the one in which it was originally shipped.

Put the RMA number on the outside of the package so that we can handle your problem
efficiently. Our service department cannot accept material received without an RMA number.

Setup

ATCA-9405 Installation and Use (6806800M71G)

57

Setup

58

ATCA-9405 Installation and Use (6806800M71G)

Packet Processor

3.1 Overview

Two independent Packet Processor Units are implemented on ATCA-9405. Each unit includes
the Packet Processor with DDR3 memory, boot flash, Ethernet interfaces, USB interfaces, serial
interfaces, power supply, and the connections to the blade infrastructure. The main functions
of the Packet Processor unit are illustrated in the Figure 3-1 on page 59.

Figure 3-1 Packet Processor Unit Overview

Chapter 3

ATCA-9405 Installation and Use (6806800M71G)

59

The functions specified in the following sections are applicable for both the Packet Processor
Units.

3.2 CN6880 Processor

ATCA-9405 design is based on Cavium OCTEON II CN6880 Multi-Core MIPS64 Processors. The
CN68XX family is targeted for high-performance, high-throughput, service-rich applications in
secure datacenter, mobile internet, and borderless enterprise applications.

The CN68XX family includes six software and pin-compatible processors, with 16 to 32
cnMIPS64 v2 cores, over 85 application acceleration engines, and real time Power Optimizer
features.

Packet Processor

High-bandwidth connectivity based on the latest standards-based SERDES I/Os including PCIe
Gen2, XAUI, DXAUI, RXAUI, SGMII, and Interlaken enables throughputs up to 40 Gbps using a
single chip or scaling to over 100 Gbps using multiple chips.

Using up to four DDR3 controllers, a 4 MB L2 Cache, and complete application acceleration,
including packet processing, Encryption/Decryption, Deep Packet Inspection (RegEx),
Compression/decompression, De-duplication, RAID, and Multi-core scaling, the CN68XX
offers both the highest compute as well as the highest throughput processing and services.

Note that the 1.5 GHz version of the processor may not be used on ATCA-9405 because of
thermal and electrical limitations. Tradeoff with regard to core count and memory size is
needed to enable usage of 1.5 GHz processor.

3.3 Cache

The CN6880 includes 37 KB of L1 instruction cache and 32 KB of L1 data cache with parity
protection and single-bit error correction for each core. The L1 caches are part of the processor
core and run at full core clock frequency. Additionally, 4 MB L2 cache is shared between all
processor cores. The L2 cache is 16-way set-associative with a 128 byte cache block, write-back
and SECDED ECC support for both the on-chip data and tags.

60

ATCA-9405 Installation and Use (6806800M71G)

3.4 System Memory

3.4.1 Memory Interface

All four CN6880 memory controllers are used on ATCA-9405. Each memory controller provides
a 72-bit (64-bit data plus 8-bit for ECC) wide DDR3 interface channel that connects with a single
DDR3 DIMM socket.

Two physical memory banks (chip select signals) are implemented per DIMM socket to allow
use of single-rank and dual-rank DIMM modules. The use of quad-rank DIMM modules is not
supported by the memory controller.

Each DDR3 memory channel runs at data rates from 600 MHz to 1333 MHz providing a total
bandwidth of up to 10.6 GBps per memory channel. The DDR3 speeds supported by CN6880
are listed in the table below.

Packet Processor

Table 3-1 DDR3 Speed Grades

Standard
Name

DDR3-800 100 MHz 10 ns 400 MHz 800 Million PC3-6400 6400 MBps

DDR3-1066 133 MHz 7.5 ns 533 MHz 1066 Million PC3-8500 8533 MBps

DDR3-1333 166 MHz 6 ns 667 MHz 1333 Million PC3-10600 10667 MBps

Memory
Clock

Cycle
Time

3.4.2 Memory Socket

Four VLP DDR3 DIMM sockets (one DIMM socket per memory channel) are provided to install
in the shelf DIMM modules. The socket is keyed for DDR3 DIMM modules using 1.5V supply
voltage.

ATCA-9405 Installation and Use (6806800M71G)

I/O Bus
Clock

Data
Transfer Per
Second

Module
Name

Peak Transfer
Rate

61

3.4.3 Memory Modules

ATCA-9405 requires VLP DDR3 DIMM modules in order to fit within the maximum component
height profile of an ATCA blade. Installed module height must not exceed 21.33 mm above PCB
surface.

The CN6880 DIMM module requirements are listed below.

Table 3-2 CN6880 Memory Module Requirements

Characteristics Value

Form Factor 240 pin Very Low Profile (VLP) DIMM

Memory Technology DDR3-1333 (PC3-10600)

DIMM Organization Single-rank or dual-rank

Device Organization x8 or x16 organized

Packet Processor

Registered DIMM

Memory Size 2 GB, 4 GB, 8 GB

Bus Width 64-bit data

8-bit ECC

Supply Voltage +1.5V

DIMMs with x4 memory devices are not supported by the memory controller.

The three preferred DIMM modules for CN6880 are listed in the table below, which provides up
to 32 GB of DDR3 main memory per Packet Processor Unit.

Table 3-3 CN6880 Preferred Memory Modules

Characteristics 2 GB DIMM 4 GB DIMM 8 GB DIMM

Vendor Smart Modular Smart Modular Smart Modular

Device SG572568EMR069P2SG SG5721288EMR069P2SD SG5721G8EMR069P2SA

Size 2 GB 4 GB 8 GB

Type DDR3-1333 DDR3-1333 DDR3-1333

CAS latency 9-9-9 9-9-9 9-9-9

62

ATCA-9405 Installation and Use (6806800M71G)

Packet Processor

Table 3-3 CN6880 Preferred Memory Modules (continued)

Characteristics 2 GB DIMM 4 GB DIMM 8 GB DIMM

Organization 18*128Mx8 (1Gb die) 18*256Mx8 (2Gb die) 18*512Mx8 (4Gb die)

Number of Ranks 2 2 2

Package 240-pin VLP Registered DIMM with ECC

Oracle Number 9706802A83 9706802A73 9706802A20

3.4.4 Thermal Sensor

Majority of DDR3 Registered DIMMs contain a Thermal Sensor on DIMM (TSOD) and its output
is tied to EVENT# pin187 on the DIMM. This signal is an active low signal and is used by the
DIMM to notify the Processor or system management device that its thermal sensor has
crossed a thermal threshold. The four thermal sensors are readable by CN6880 and by the
IPMC. The EVENT# interrupts are connected to CN6880 (which is responsible for interrupt
routing) GPIO pins.

3.5 Octeon U-Boot

The two Octeon-II packet processors on the ATCA-9405 do not have a boot flash device. They
are started using the oct-remote-boot command from the Octeon SDK via the PCI bus
from the Freescale P2020 service processor.

Both Packet Processor Units are not powered when payload power is enabled by the IPMC. The
Packet Processor Units remain in power-off state until enabled by the Service Processor.

The CN6880 processors are hardware strapped for PCI Express (PCIe) Remote Boot Mode. After
power up or reset the processor cores remain in idle state until they are booted via PCIe
interface by the Service Processor.

3.5.1 NVRAM

U-Boot for the Octeon processors does not use an EEPROM to store its environment
parameters. Instead, the configuration parameters for U-Boot are transferred together with
the boot image from the service processor.

ATCA-9405 Installation and Use (6806800M71G)

63

Application specific parameters can be kept on the service processor. It is not possible to save
environment parameter changes within U-Boot as the saveenv command does not work.

3.5.2 Network Interfaces

The following network interfaces are available in U-Boot:

 octmgmt0 (front-panel management interface)

 octeth0 (1000BaseX interface to switch)

 octeth4 (DXAUI interface 0 to switch)

 octeth5 (DXAUI interface 1 to switch)

3.6 SerDes Configuration

Packets and control information can flow from the CN6880 via any of the SGMII, XAUI, DXAUI,
RXAUI, Interlaken, or PCIe interfaces. Internally CN6880 supports five SerDes quad-lane
modules (QLMs) for a total of 20 SERDES lanes.

Packet Processor

64

For the default configuration of ATCA-9405 (where the Packet Processors are interconnected
via Interlaken Interface), one PCI Express, one Interlaken, two DXAUI and two RXAUI interfaces
are provided.

Note that ATCA-9405-ILK-MODULE must be assembled on this version.

The corresponding QLM mode strapping is listed in the table below. The QLM reference clock
must be configured according to the IO interface standard used.

Table 3-4 CN6880 SerDes Configuration (Interlaken Mode)

QLM Supported Interfaces Configuration

QLM0 XAUI, RXAUI, SGMII RXAUI

156.25 MHz

QLM1 PCI-Express, Interlaken Interlaken

156.25 MHz

QLM2 PCI-Express, Interlaken XAUI,

DXAUI, SGMII

DXAUI

156.25 MHz

ATCA-9405 Installation and Use (6806800M71G)

Table 3-4 CN6880 SerDes Configuration (Interlaken Mode) (continued)

QLM Supported Interfaces Configuration

QLM3 PCI-Express XAUI, SGMII PCI Express Gen 1

QLM4 PCI Express XAUI, DXAUI, SGMII DXAUI

3.7 PCI Express Interface

CN6880 integrates two PCIe Interfaces that are compliant with the PCIe Base Specification
Revision 2.0. They are configured at boot time to act as either root complex or endpoint. The
PHY of the PCIe interface supports transmission rate of up to 5.0 Gbps per lane. The two
interfaces can be configured as one single or two independent ports with x1, x2, x4, or x8 link
widths.

Packet Processor

100 MHz

156.25 MHz

On ATCA-9405, PCIe Interface 0 (using QLM3 SerDes module) is configured to x1 link width and
is connected to the PLX PEX8608 switch. For more details regarding PCIe infrastructure, refer
to Service Infrastructure on page 101.

PCIe Interface 1 is not used on ATCA-9405.

3.8 Ethernet Interface

The CN6880 provides a number of integrated on-chip Ethernet controllers and supports
various interfaces standards. The CN6880 family can implement up to:

 16 SGMII/1000BASE-X SerDes interfaces through up to 4 four-port (four-lane) packet

interfaces. A full-duplex port consists of four external pins, a differential output pair and a
differential input pair. CN6880 couples logic that implements the SGMII and/or 1000BASE­X protocols on SerDes lanes with a 10/100/1000 802.3 MAC.

 Five XAUI SerDes interfaces in up to 4 four-lane packet interfaces. Each interface consists

of 16 external pins in total, four differential output pairs plus four differential input pairs.
CN6880 couples logic that implements the XAUI, reduced XAUI (RXAUI) or double data
rate XAUI (DXAUI) interface/protocols on SerDes lanes with an IEEE 802.3-2005 MAC.

ATCA-9405 Installation and Use (6806800M71G)

65

In addition, CN6880 provides a 10/100/1000 Ethernet MAC-to -PHY interface supporting either
an MII MAC or an RGMIII MAC. The MII links are fully compliant with the IEEE 802.3
specifications, and the RGMII links are fully compliant with the HP RGMII 1.3 specifications.

On ATCA-9405 two DXAUI, two RXAUI, and one RGMII interface are implemented.

3.8.1 Front Panel Interface

The RGMII interface is connected to a 10/100/1000Base-T port at the front panel. The physical
interface is implemented using Marvell 88E1512 Single Port Gigabit Ethernet Transceiver and
an RJ45 connector with integrated magnetics.

3.8.2 Base and Fabric Interface

The SGMII interface (using QLM0 SerDes module) is connected to the Ethernet Switch and
provides a Gigabit Ethernet interface to the control plane (Base Interface).

Packet Processor

The two DXAUI interfaces (using QLM2 and QLM4 SerDes module) are connected to the
Ethernet Switch and provide a 2x20G Ethernet interface to the data plane (Fabric Interface).

The two RXAUI interfaces (using QLM0 SerDes module) are connected to the Ethernet Switch
and provide a 20 Gigabit Ethernet interface to the data plane (FI).

3.9 Interlaken Interface

The Interlaken interface unit provides a narrow, high-speed, channelized packet interface using
the scalable Interlaken protocol. It conforms to the Interlaken Protocol Definition V1.2 and the
Interlaken Look-Aside Protocol Definition V1.1. Interlaken combines the channelization,
programmable burst sizes, and per-channel backpressure features of the SPI4.2 protocol with
the long-reach and reduced pin count of the XAUI protocol, and it offers the ability to tailor the
design to the interface capacity of the application.

The Interlaken interface unit supports SerDes lanes running at up to 6.25 Gbps, two
links/interfaces, and x8, x4, x2, and x1 configurations. On ATCA-9405 the Interlaken interface
(using QLM1 SerDes module) is used for interconnection between the two Packet Processors
(when ATCA-9405-ILK-MODULE is assembled).

66

ATCA-9405 Installation and Use (6806800M71G)

3.10 USB Interface

The CN6880 USB subsystem contains an EHCI host controller, an OHCI host controller, and two
physical USB ports. The USB interface is with USB specification, Revision 1.1 and supports high­speed (480 Mbps), full-speed (12 Mbps), and low-speed (1.5 Mbps) operation. For more
information refer to the Cavium OCTEON II Hardware Reference Manual.

The USB interface 1 is not used on ATCA-9405.

An assembly option (footprint for a high speed USB switch type FSUSB31K8X) to allow
connection of USB Interface 0 toward ARTM is available. If you require this, then contact the
sales/marketing team.

3.11 UART Interface

Packet Processor

The CN6880 provides a dual UART compatible with the PC16550D. Both interfaces consist of
four wires RXD, TXD, RTS, and CTS. The interfaces run in full duplex mode and the baud rates
are software programmable.

The first interface COM0 is connected to the Glue Logic FPGA that routes the interface to either
the front panel connector or to the Terminal server. The connection to the front panel console
connector is implemented via RS-232 transceiver. All four signals (RXD, TXD, RTS, and CTS) are
connected to an RJ45 connector using Cisco DTE pin out assignment. Serial port parameters for
the front panel interface are 9600 baud, 8 data bits, no parity bit, and 1 stop bit.

When serial port COM0 is routed to the Terminal Server for console redirection into the base
network, only TXD and RXD signals are used. IPMC can control serial port routing. Serial port
parameters for this interface are 9600 baud, 8 data bits, no parity bit, and 1 stop bit. For more
information, see Serial Line Selection on page 139.

The second interface COM1 (only signals TXD and RXD) is connected to a standard 16-pin
header for debugging purposes.

ATCA-9405 Installation and Use (6806800M71G)

67

3.12 I2C Interface

The CN6880 has 2 two-wire serial interfaces (TWSI) that provide multi-master and master­slave I2C mode support. TWSI interface 0 is connected to the four DIMM modules for SPD
detection. TWSI interface 1 is connected to the external EEPROM and to the Glue Logic FPGA
for access to internal status and control registers.

Table 3-5 CN6880 I2C Bus Assignment

Address TWSI Component Function

0xA0 0 NA DIMM1 Memory Module SPD PROM

0xA2 0 NA DIMM2 Memory Module SPD PROM

0xA4 0 NA DIMM3 Memory Module SPD PROM

0xA6 0 NA DIMM4 Memory Module SPD PROM

0xA0 1 24LC128 U-Boot Parameter Storage EEPROM

Packet Processor

0xFE 1 NA Glue Logic FPGA

3.13 JTAG Interface

The IEEE 1149.1 compliant JTAG boundary scan interface of CN6880 is connected to an
onboard 16-pin header to support a processor emulator for board debug. The additional
signals for MIPS EJTAG support according to EJTAG Specification Revision 5.0 are supported.
For connector pin out details, refer to Connector Pin Assignment.

3.14 Interrupts

Each Packet Processor is responsible for handling of interrupts that are generated within its
own domain.

3.14.1 Packet Processor Interrupts

The CN6880 includes a central interrupt unit that allows centralized collection of internal and
external interrupts as well as interrupt selection and distribution to the cores and to the
integrated PCI Express controllers.

68

ATCA-9405 Installation and Use (6806800M71G)

Packet Processor

Up to 16 GPIO pins are provided by the CN6880 to connect external interrupts with the central
interrupt unit. Each GPIO pin can be programmed to be a level-sensitive interrupt pin or an
edge-triggered interrupt pin.

All interrupt sources from the packet processor domains are directly connected to the
GPIO[5:1] inputs of the associated Packet Processor.

The following table lists all interrupt sources of the packet processor domains and how the
interrupts are routed on the board:

Table 3-6 Packet Processor Interrupts

Interrupt Source Interrupt Name Usage IRQ Line

PP1 PP1_DDR0_EVENT_N DIMM1 Thermal Event PP1_GPIO2

PP1_DDR1_EVENT_N DIMM2 Thermal Event PP1_GPIO3

PP1_DDR2_EVENT_N DIMM3 Thermal Event PP1_GPIO4

PP2 PP2_DDR0_EVENT_N DIMM1 Thermal Event PP2_GPIO2

3.15 Power Supply

ATCA-9405 supports software controlled power-down and power-on sequence for the two
Packet Processor units as described in Hot Plug Support on page 102.

PP1_DDR3_EVENT_N DIMM4 Thermal Event PP1_GPIO5

PP1_ETH_INT_N 88E1512 PHY Interrupt PP1_GPIO1

PP2_DDR1_EVENT_N DIMM2 Thermal Event PP2_GPIO3

PP2_DDR2_EVENT_N DIMM3 Thermal Event PP2_GPIO4

PP2_DDR3_EVENT_N DIMM4 Thermal Event PP2_GPIO5

PP2_ETH_INT_N 88E1512 PHY Interrupt PP2_GPIO1

ATCA-9405 Installation and Use (6806800M71G)

69

3.16 Cooling

A passive heat sink is mounted on top of the CN6880 package. The heat sink is designed to
withstand shock and vibration tests according to the environmental conditions. The heat sink
keeps the processor die temperature below the maximum rating of 125°C under any
conditions listed in Environmental Considerations on page 53.

Packet Processor

70

ATCA-9405 Installation and Use (6806800M71G)

Service Processor

4.1 Overview

The Service Processor (SP) functional unit is in charge of blade and RTM infrastructure. This
includes management and control of the two CN6880 Packet Processors, the 98CX8234
Ethernet Switch, the PEX8608 PCI e Switch, and the RTM. The SP also takes the root complex
role for the complete PCIe infrastructure on the blade and RTM. The main functions of the SP
are shown in the Figure 4-1 on page 71.

Figure 4-1 Service Processor Unit Overview

Chapter 4

ATCA-9405 Installation and Use (6806800M71G)

71

4.2 P2020 Processor

A Freescale P2020 QorIQ Communication Processor is used on ATCA-9405 as the onboard SP.
The processor is manufactured in 45nm process technology and combines dual Power
Architecture™ e500v2 processor cores with system logic required for networking, wireless
infrastructure, and telecommunications applications.

The P2020 is available with three speed grades running at 800 MHz, 1000 MHz, and 1200 MHz
core clock frequency. All speed grades are supported, the default speed grade on ATCA-9405
is 1.0GHz. Additionally, the board is prepared to support all three speed grades of P2010 single
core derivative.

4.3 Cache

The P2020 includes 32 KB of L1 instruction cache and 32 KB of L1 data cache with parity
protection for each core. L1 caches can be locked entirely or on a per-line basis, with separate
locking for instructions and data. The 512 KB L2 cache is common to both processors and has
full ECC protection on 64-bit boundary and supports instruction caching, data caching, or both
modes. The L1 and L2 caches are part of the P2020 core complex and run at full core clock
frequency.

Service Processor

4.4 Main Memory

4.4.1 Memory Interface

The P2020 memory controller on ATCA-9405 is configured for DDR3 SDRAM mode. The
controller provides a 72-bit (64-bit data plus 8-bit for ECC) wide DDR3 interface (channel) that
connects the P2020 with a single DDR3 DIMM socket.

Two physical memory banks (chip select signals) are implemented on DIMM socket to allow
use of single-rank and dual-rank DIMM modules.

72

ATCA-9405 Installation and Use (6806800M71G)

Service Processor

The DDR3 memory interface runs at a maximum frequency of 400 MHz providing a total
bandwidth of 6.4 GBps. The fastest supported DDR3 speed is DDR3-800 (PC3-6400).

Table 4-1 P2020 DDR3 Speed Grades

Standard
Name

DDR3-800 100 MHz 10 ns 400 MHz 800 Million PC3-6400 6400 MBps

DDR3-1066 133 MHz 7.5 ns 533 MHz 1066 Million PC3-8500 8533 MBps

DDR3-1333 166 MHz 6 ns 667 MHz 133 Million PC3-10600 10667 MBps

Memory
Clock Cycle Time

4.4.2 Memory Socket

A Very Low Profile (VLP) DDR3 DIMM socket is provided to install of-the-shelf DIMM modules.
The socket is keyed for DDR3 DIMM modules using 1.5V supply voltage.

4.4.3 Memory Modules

ATCA-9405 requires VLP DDR3 DIMM modules in order to fit within the maximum component
height profile of an ATCA blade. Installed module height must not exceed 21.33 mm above PCB
surface.

Table 4-2 P2020 Memory Module Requirements

Characteristics Value

Form Factor 240 pin Very Low Profile (VLP) DIMM

I/O Bus
Clock

Data Transfers
per second

Module
Name

Peak Transfer
Rates

Memory Technology DDR3-800 (PC3-6400)

Registered DIMM

DIMM Organization Single-rank or dual-rank

Device Organization x8 or x16 organized

Memory Size 2 GB, 4 GB, 8 GB

Bus Width 64-bit data

8-bit ECC

Supply Voltage +1.5V

ATCA-9405 Installation and Use (6806800M71G)

73

Service Processor

DIMMs with x4 memory devices are not supported by the memory controller.

Default main memory size is 2 GB using one Smart Modular SG572568EMR069P2SG DIMM
module. The three preferred DIMM modules for P2020 are listed in the table below, providing
up to 8 GB of DDR3 main memory.

Table 4-3 P2020 Preferred Memory Modules

Characteristics 2 GB DIMM 4 GB DIMM 8 GB DIMM

Vendor Smart Modular Smart Modular Smart Modular

Device SG572568EMR069P2SG SG5721288EMR069P2SD SG5721G8EMR069P2SA

Size 2 GB 4 GB 8 GB

Type DDR3-1333 DDR3-1333 DDR3-1333

CAS latency 9-9-9 9-9-9 9-9-9

Organization 18*128Mx8 (1Gb die) 18*256Mx8 (2Gb die) 18*512Mx8 (4Gb die)

Number of Ranks 2 2 2

Package 240-pin VLP Registered DIMM with ECC

Oracle Number 9706802A83 9706802A73 9706802A20

4.4.4 Persistent Memory

The P2020 offers both hardware and software options to support persistent main memory.
The persistent memory is an array of random access memory that preserves its contents
during a warm/soft reset. Persistent memory is a necessary prerequisite for performing post
mortem analysis of log data after reset and reboot of the payload CPU.

For persistent memory support on ATCA-9405, a dedicated register is implemented in the Glue
Logic FPGA to enable or disable persistent memory by software. If persistent memory is
enabled, the contents of the main memory stay unchanged after any applied reset, except
power-up reset.

After reset is released and the board is booted, the application must check the persistent
memory bit in FPGA and (if set) initialize the main memory but keep persistent memory area
untouched.

74

ATCA-9405 Installation and Use (6806800M71G)

4.4.5 Thermal Sensor

Majority of DDR3 Registered DIMMs contain a Thermal Sensor on DIMM (TSOD) and its output
is tied to EVENT# pin187 on the DIMM. This signal is an active low signal and is used by the
DIMM to notify the Processor or system management device that its thermal sensor has
crossed a thermal threshold. The thermal sensor is readable by P2020 and by the IPMC. The
EVENT# interrupt is connected to the FPGA (which is responsible for interrupt routing).

4.5 SP U-Boot

After power up or reset, the following necessary steps are taken by P2020 processor:

 boot from active SPI Boot Flash

 initialize PCIe infrastructure

 initialize network infrastructure

Service Processor

 power-up Packet Processors

 initialize PCIe interfaces of Packet Processors

 boot Packet Processors via PCIe

4.5.1 Environment Variables

U-Boot uses environment variables to both control the behaviour of various U-Boot
components and to report information to user applications.

The working set of the U-Boot environment variables is stored in memory and can be accessed
using the setenv and getenv commands. The values of environment variables can be:

 The content of the NVRAM (an I2C EEPROM), or (if the NVRAMs CRC is invalid) a set of

compiled-in defaults.

 The content of the IPMI boot parameter storage

 Dynamic variables set during the boot-up phase

 Variables set by the user via the U-Boot shell.

ATCA-9405 Installation and Use (6806800M71G)

75

The above listing basically describes the order of precedence, that is a variable stored in the
NVRAM can potentially be overridden, modified, or deleted by a variable of the same name
stored in the IPMI boot parameter storage, and later on by dynamic variables declared during
the boot process or by the user from the shell prompt.

Although the saveenv command stores the complete set of variables currently being used
into the NVRAM, the actual setting of these variables might not become valid because they are
overridden by the other entities.

Note that the U-Boot does not synchronize the NVRAM storage and the IPMI boot parameter
storage by itself.

If the NVRAMs CRC is invalid, the NVRAM is initialized with the blade's default values
immediately after detecting the bad CRC (before reading additional values from the IPMC).

4.5.2 Passing Parameter Set to the Operating System

Service Processor

An operating system making use of environment variables should only rely on the RAM copy of
the parameter set.

U-Boot passes its current parameters via the firmware device tree. It generates a node /u-
boot-env, containing a node for each environment variable. This is done by the bootm
command as the final step before executing the OS image. Under Linux, these parameters
appear under the directory /proc/dev-tree/u-boot-env.

4.5.3 Dynamic Variables Set During the Boot Phase

This section lists the variables that are set during the boot phase.

Table 4-4 Dynamic Variables Set During the Boot Phase

Variable Name Description

boottime The time (UNIX time) when the blade has started. If this variable is set

to "-1" then the RTC does not run at startup

physical_slot The physical slot number where the blade is mounted

logical_slot The logical slot number where the blade is mounted

bootbank The physical boot bank number from which the blade has started (0 or

1)

76

ATCA-9405 Installation and Use (6806800M71G)

Service Processor

Table 4-4 Dynamic Variables Set During the Boot Phase (continued)

Variable Name Description

shelf_id The first byte of the shelf identifier as read from the shelf manager

shelf_location Two bytes encoding the logical slot number and the 1st byte of the shelf

ID, each octet separated by a colon

dhcp_clid The DHCP client ID (option 61) used in DHCP discover

uboot_version The U-Boot firmware version (in the format V<x>.<y>.<z>).

ethaddr
eth1addr
eth2addr

fpga_version Version of Glue Logic FPGA

The Ethernet MAC addresses for the local network interfaces

4.5.4 Variables for Controlling the Boot Progress

Table 4-5 Variables for Controlling the Boot Progress

Variable Name Default Description

bmc_wd_timeout undefined Configures the IPMI watchdog time-out for booting

the OS in seconds. Setting it to -1 (default) disables
the IPMI watchdog.

ethdefault undefined The default network interface to be configured. If

set, the ethact variable is initially assiged to the
value of this parameter.

bootcmd run $ramboot A sequence of commands executed to boot the

operating system (via auto boot or the boot
command)

bootargs undefined Arguments for the operating system (that is, the

linux kernel command line or the vxWorks boot
parameters).

bootdelay 10 Delay in seconds before the automatic boot

serverip undefined The IP address of the boot server

ATCA-9405 Installation and Use (6806800M71G)

sequence starts.
A value of "-1" disables autoboot

77

Table 4-5 Variables for Controlling the Boot Progress (continued)

Variable Name Default Description

ipaddr undefined The IP address used for network communication

gatewayip undefined The default gateway IP

pram 0 This is a decimal value encoding a memory size in

4.5.5 Firmware Update

The command firmware is used to show the versions of various components and to upgrade
some of them.

Service Processor

kilobytes (1024 bytes). The specified amount of
memory is reserved at the end of physical memory
and will not be overwritten by U-Boot. This
memory will also be retained over reset.

The size of usable memory is reported in the
variable "pram_size".

 The show command displays version information

 The update command requires a valid firmware image in memory. It supports the

following component:

– U-Boot for P2020

The tool always programs the standby image and after successful update, activates it
to be used at the next reboot.
It also activates the failsafe logic. It is in the responsibility of the end-user to deactivate
the fail-safe logic once the boot sequence is successful. The fail-safe logic can be
deactivated via IPMI using the OEM command set feature configuration as
described in Set Feature Configuration on page 151.

Programming a firmware image requires that the image is present as .fri file. The firmware
update command will verify that the contents of the fri-file are suitable to be programmed into
the selected device.

4.5.6 Application/OS Boot

The default boot device of the ATCA-9405 is the internal USB disk.

78

ATCA-9405 Installation and Use (6806800M71G)

4.5.6.1 Default Boot Sequences

Various default boot sequences are stored in environment variables. These variables can be
executed as script using the run command (for example, run nfsboot).

4.5.6.1.1 Disk (bootcmd)

This is the default boot operation (bootcmd). The boot command is run usbboot

4.5.6.1.2 TFTP/NFS (nfsboot)

Load the kernel $bootfile via TFTP from the nfs server $serverip, path $rootpath, and boot it
using $serverip$rootpath as nfsroot, and $netdev as linux network device.

The boot command is run nfsboot

Netdev value is eth0 for front panel connector MGT ETH

Service Processor

4.5.7 Memory/Address Map Initialization

4.5.7.1 Address Map

U-Boot configures the P2020 address map via local access windows (LAWs) and MMU mapping
entries (TLBs) as described in the following table:

Table 4-6 Address Map

Device Physical Address Range Virtual Start Address Note

DDR3 0x0.0000.0000

0x0.xxxx.xxxx

PCI1 Memory 0xc.0000.0000 0x8000.0000 1GB Non-prefetchable

PCI1 I/O 0xf.ffc0.0000 0xffc0.0000 256KB I/O space

FPGA 0xf.ef00.0000 0xef00.0000 128 bytes FPGA register

P2020 CCSR 0xf.ffe0.0000 0xffe0.0000 P2020 CCSR space

0x0000.0000 2GB or 4GB DDR3

memory

set

ATCA-9405 Installation and Use (6806800M71G)

79

4.6 Local Bus

The P2020 includes an Enhanced Local Bus Controller (eLBC). The main component of the eLBC
is its memory controller, which provides a seamless 16-bit interface to many types of memory
devices and peripherals. The memory controller is responsible for controlling eight memory
banks (chip selects) shared by a general-purpose chip select machine (GPCM), an FCM, and up
to three UPM machines. The eLBC offers a multiplexed 16-bit address and data bus operating
at up to 83 MHz. Data checking and protection features, such as parity support and write
protection are included.

The eLBC supports ratios of 4, 8, and 16 between the faster internal CCB clock and slower
external bus clock. The selected divider for ATCA-9405 is 16, resulting in a local bus frequency
of 31 MHz at a platform frequency of 500 MHz (1000 MHz processor).

The eLBC provides one GPCM, one FCM, and three UPMs for the local bus to allow the
implementation of memory systems with very specific timing requirements. The GPCM
provides interfacing for simpler, lower-performance memories, and memory-mapped devices.
It has inherently lower performance because it does not support bursting. For this reason,
GPCM-controlled banks are used primarily for boot-loading from NVRAM or NOR Flash, and
access to low-performance memory-mapped peripherals.

Service Processor

On ATCA-9405, the GPCM provides the local bus interface to the FPGA. The FCM and UPM
controllers are not used. The FPGA is connected to chip select LCS3# and configured as 8-bit
device.

Contact sales/marketing team, if you require a detailed CPLD and FPGA specification.

4.7 SerDes Configuration

The P2020 provides four SerDes blocks that support the SGMII, serial RapidIO, and PCIe high­speed I/O interface standards. Each SerDes can be mapped to one of the different IO interfaces
depending on power-on reset (POR) configuration.

80

ATCA-9405 Installation and Use (6806800M71G)

Service Processor

Two SGMII and one PCIe interface are needed for ATCA-9405. The corresponding SerDes
configuration is listed in the table below. The SerDes reference clock for this configuration must
be 100 MHz.

Table 4-7 P2020 SerDes Configuration

Reset Configuration Value SerDes Configuration

cfg_IO_ports[0:3] 1110 SerDes Lane 0: PCI Express 1 (x1, 2.5 Gbps)

SerDes Lane 1: PCI Express 2 (x1, 2.5 Gbps) - not used
SerDes Lane 2: SGMII eTSEC2 (x1, 1.25 Gbps)
SerDes Lane 3: SGMII eTSEC3 (x1, 1.25 Gbps)

cfg_sgmii2 0 eTSEC2 operates in SGMII mode

cfg_sgmii3 0 eTSEC3 operates in SGMII mode

SerDes Lane 1 is not used on ATCA-9405, so it is left unconnected.

4.8 PCI Express Interface

The P2020 supports three PCIe interfaces that are compliant with the PCIe Base Specification
Revision 1.0a. They are configued at boot time to act as either root complex or endpoint. The
PHY of the PCIe interface operates at a transmission rate of 2.5 Gbps (data rate of 2 Gbps) per
lane. The ports can be configured for x1, x2, or x4 link widths.

On ATCA-9405, PCIe Port 1 (SerDes Lane 0) is configured to x1 link width and is connected to
the PLX PEX8608 switch. P2020 acts as root complex and is responsible for blade control and
monitoring as well as PCIe hot plug handling. The remaining two PCIe Ports are not used.

For more details regarding PCIe infrastructure and hot plug handling, refer to Service

Infrastructure on page 101.

ATCA-9405 Installation and Use (6806800M71G)

81

4.9 Ethernet Interface

The P2020 has three on-chip enhanced three-speed Ethernet controllers (eTSECs). The eTSECs
incorporate a media access control (MAC) sub layer that supports 10/100/1000 Mbps
Ethernet/802.3 networks with SGMII, GMII, RGMII, MII, RMII, TBI, and RTBI physical interfaces,
as well as 8-bit or 16-bit FIFO interfaces that bypass the Ethernet MAC.

The eTSECs include 2 KB receive and 10 KB transmit FIFOs and DMA functions, support for
programmable CRC generation and checking, RMON statistics, jumbo frames, TCP/IP
acceleration and QoS features, VLAN insertion and deletion and MAC address recognition.

On ATCA-9405, eTSEC1 is configured to RGMII mode while eTSEC2 and eTSEC3 are configured
to SGMII mode. The corresponding strapping is listed in the table below.

Table 4-8 P2020 eTSEC Configuration

Reset Configuration Value eTSEC Configuration

Service Processor

cfg_tsec_reduce
cfg_tsec1_prtcl[0:1]

cfg_sgmii2 0 eTSEC2 operates in SGMII mode

cfg_sgmii3 0 eTSEC3 operates in SGMII mode

0
10

4.9.1 Front Panel Interface

P2020 eTSEC1 interface is connected to a 10/100/1000Base-T port at the front panel. The
physical interface is implemented using Marvell 88E1512 Single Port Gigabit Ethernet
Transceiver (connected to the P2020 in RGMII mode) and an RJ45 connector with integrated
magnetics.

4.9.2 Base and Fabric Interface

The eTSEC2 and TSEC3 interfaces are connected to the Ethernet Switch and provide a Gigabit
Ethernet interface to both the control plane (base interface) and the data plane (fabric
interface). The two eTSECs are configured for SGMII mode and are directly connected to the
Ethernet Switch.

eTSEC1 operates in RGMII mode

82

ATCA-9405 Installation and Use (6806800M71G)

4.10 SPI Interface

The P2020 includes a full duplex four-wire SPI interface. The SPI interface can support up to four
separate SPI devices. The P2020 SPI interface connects to the SPI Boot Flashes, to the Telecom
DPLL and to the Terminal Server. The following table shows the P2020 chip select assignments
for the SPI interface.

Table 4-9 P2020 SPI Chip Select Assignment

SPI Chip Select Value

CS0# SPI Boot Flash 1 (default)

CS1# SPI Boot Flash 2 (backup)

CS2# Telecom DPLL

CS3# Terminal Server

Service Processor

4.10.1 Boot Flash

Two SPI Flash devices are provided as P2020 boot devices.

Hardware write protection signal WP# of SPI Flash is pulled down permanently to enable the
hardware protection built in the SPI Flash. Individual Boot Flash sector write protection must be
implemented via software mechanism according to SPI Flash specification.

4.10.2 Boot Flash Selection

For crisis recovery, Boot Flash 1 (default) and Boot Flash 2 (recovery) can be exchanged under
IPMC software control through logic implemented in the FPGA.

ATCA-9405 Installation and Use (6806800M71G)

83

Service Processor

By default, the P2020 boots from Boot Flash 1. An IPMI OEM command can be used to send a
message to the IPMC to change the boot device. The IPMC provides an IPMI sensor to control
the signal BOOT_SELECT. If the BOOT_SELECT signal is set high, the payload processor boots
from Boot Flash 2 after reset.

Table 4-10 P2020 Boot Flash Selection

SW1-3
Boot from

debug header

OFF OFF - LOW Selected

OFF OFF - HIGH Selected

OFF ON ON - Selected

OFF ON OFF - Selected

ON - - - Selected

SW1-1
IPMC or manual

boot selection

4.11 USB Interface

The P2020 provides an USB 2.0 compliant controller which can be configured to operate as a
stand-alone host or stand-alone device. The controller is enhanced host controller interface
(EHCI) compatible and supports high-speed (480 Mbps), full-speed (12 Mbps), and low-speed
(1.5 Mbps) operation. The USB interface requires an external PHY with a UTMI+ low pin count
interface.

The USB PHY is configured as host controller and connects to an USB hub in order to provide
multiple USB ports. A high-speed USB 2.0 hub controller is used. The device offers two
downstream ports supporting high-speed, full-speed, and low-speed operation.

The hub provides fully integrated USB termination and pull-up/pull-down resistors, over
current protection and power supply for the USB ports. USB port 1 is connected to an USB
connector at the front panel. USB port 2 is connected to an on-board eUSB Flash Drive.

SW1-2
Manual SPI

Flash selection

IPMC
BOOT_
SELECT

SPI Flash 0
(Default)

SPI Flash 1
(Recovery)

Debug
Header

4.11.1 USB Connector

One standard USB type A connector is provided at the face plate for USB access to the SP. For
connector pin out details, refer to USB Port on page 39.

84

ATCA-9405 Installation and Use (6806800M71G)

4.11.2 e-USB Flash Drive

ATCA-9405 provides a factory option for eUSB Flash Drives. The Flash Drive can be used for local
mass storage, persistent user data storage or booting of operating system and user images. A
low profile 2mm eUSB header on the board is connected to the P2020 to support eUSB
modules from various vendors.

The default assembly SG9ED52M1GG4NEMR from Smart Modular Technology provides 16 GB
of User Flash memory. The module provides sustained read speeds of up to 35 MBps and writes
up to 23 MBps. For connector pin-out and footprint, refer to Table "eUSB Drive Connector Pin-

out" on page 42.

4.12 UART Interface

The P2020 provides a dual UART compatible with 16450 and the PC16550D. Both interfaces
consist of four wires RXD, TXD, RTS, and CTS. The interfaces run in full duplex mode and the
baud rates are software programmable.

Service Processor

The first interface COM0 is connected to the FPGA that is able to route the interface to either
the front panel connector and/or to the Terminal server. The connection to the front panel
console connector is implemented via RS-232 transceiver. All four signals (RXD, TXD, RTS, and
CTS) are connected to an RJ45 connector. For connector pin out, refer to Serial Console Ports.
Serial port parameters for the front panel interface are 9600 baud, 8 data bits, no parity bit, and
1 stop bit.

When serial port COM0 is routed to the Terminal Server for console redirection into the base
network, only TXD and RXD signals are used. Serial port parameters for this interface are 9600
baud, 8 data bits, no parity bit, and 1 stop bit.

The second interface COM1 (only signals TXD and RXD) is connected to a standard 16-pin
header for debugging purposes.

4.13 I2C Interface

The P2020 includes a dual I2C controller with open-drain two-wire interfaces that provides
multi-master and master-slave I2C mode support.

ATCA-9405 Installation and Use (6806800M71G)

85

I2C bus 1 is used to connect to a boot sequencer memory and the SPD PROM device of the
DDR3 memory module. The boot sequencer can be optionally used for initialization of the
P2020.

I2C bus 2 is connected to an onboard RTC (DS1337).

Table 4-11 P2020 I2C Bus Assignment

Address Bus Component Function

0xA0 1 AT24C64C Boot Sequencer ROM

0xA2 1 SPD DIMM Module SPD PROM

0xD0 2 DS1337U+ Real Time Clock

4.13.1 Real Time Clock (RTC)

The blade provides an I2C™-bus-compatible real-time clock type DS1337 from Maxim. This
device contains a real-time clock/calendar and 31 bytes of static random access memory
(SRAM). The real-time clock/calendar provides seconds, minutes, hours, day, date, month, and
year information. The end of the month date is automatically adjusted for months with fewer
than 31 days, including corrections for leap year up to the year 2100. The clock operates in
either the 24hr or 12hr format with an AM/PM indicator. A battery socket is provided on the
board to backup the RTC power supply.

Service Processor

4.14 JTAG Interface

The IEEE 1149.1 compliant JTAG boundary scan interface of the P2020 is connected to an
onboard 16-pin COP header to support a processor emulator for board debug. For connector
pin out details, refer to Connector Pin Assignment.

4.15 Interrupts

The Service Processor is responsible for handling of infrastructure related interrupts.

86

ATCA-9405 Installation and Use (6806800M71G)

4.15.1 Service Processor Interrupts

The P2020 provides a programmable interrupt controller with up to twelve interrupt request
inputs with programmable polarity and sense of each signal. Only interrupt inputs IRQ[6:0] are
used on ATCA-9405.

The following table lists all interrupt sources of the Service Processor domain and how the
interrupts are routed on the board.

Table 4-12 Service Processor Interrupts

Interrupt Source Interrupt Name Comments IRQ Line

Service Processor

P2020 SP_INT_OUT# Persistent Memory

PEX8608 PEX_INTA_N IRQ[6:0]

88X2241 FAB_CH1_LASI[3:0]_N Link Alarm Status for

ACS8525 LCCB_INTREQ_N IRQ[6:0]

98CX8234 SW_INT[3:0]_N IRQ[6:0]

88E1322 BASE_PHY_INT_N IRQ[6:0]

88E1512 SP_ETH_INT_N IRQ[6:0]

4.16 Cooling

A passive heat sink is mounted on top of the P2020 package. The heat sink is designed to
withstand shock and vibration tests according to the environmental considerations. The heat
sink keeps the processor die temperature below the maximum rating of 125°C under any
conditions listed in Environmental Considerations on page 53.

IRQ[6:0]

Interrupt (IRQ_OUT#)

SP_DIMM_EVENT_N DIMM Thermal Event IRQ[6:0]

SP_RTC_IRQ_N Real Time Clock Interrupt IRQ[6:0]

PEX_FATA_ERR_N IRQ[6:0]

IRQ[6:0]

Fabric Channel 1

FAB_CH2_LASI[3:0]_N Link Alarm Status for

Fabric Channel 2

IRQ[6:0]

ATCA-9405 Installation and Use (6806800M71G)

87

Service Processor

88

ATCA-9405 Installation and Use (6806800M71G)

Ethernet Infrastructure

5.1 Overview

The Ethernet infrastructure on ATCA-9405 provides switching between any on-board Ethernet
port, the ATCA base and fabric interface, and the Ethernet ports provided through RTM.
Separate networks for control and data planes are implemented using VLAN and port based
filters, inside the Ethernet switch device.

Table 5-1 Data Plane, Control Plane, and Other Interfaces

Data Plane (DP) Control Plane (CP) Others

Chapter 5

40 GbE connection to
redundant Fabric Interface

60 GbE connection to each
Packet Processor (PP)

1 GbE connection to Service
Processor (SP)

160 GbE connection to RTM Dual 1 GbE connection to RTM

20 GbE connection to Update
Channel

1 GbE connection to redundant
base interface

1 GbE connection to each
Packet Processor (PP)

1 GbE connection to Service
Processor (SP)

1 GbE connection from each
Packet Processor (PP) to Face
Plate

1 GbE connection from Service
Processor (SP) to Face Plate

ATCA-9405 Installation and Use (6806800M71G)

89

Figure 5-1 Ethernet Infrastructure

Ethernet Infrastructure

90

Double data rate XAUI (DXAUI) is 20Gbps interface between Marvell Switch and Cavium CPU,
using four SerDes lanes running at 6.25Gbps.

ATCA-9405 Installation and Use (6806800M71G)

5.2 Ethernet Switch

The Ethernet switching device used on ATCA-9405 is Marvell 98CX8234, a highly integrated
32-Port 10 Gigabit Ethernet L2+ Packet Processor with 40 Gigabit Ethernet Uplinks.

The 98CX8234 is a member of the Prestera®-CX family that addresses bandwidth demand,
high density 10G solutions, and support for future 40G upgradability. The device is a packet
processor that is managed with an external CPU connected through PCI Express interface. The
device supports wide and flexible configuration of network interfaces from 1 GbE to 40 GbE
ports.

The ports can operate in Serial Gigabit Media Independent Interface (SGMII), 10 Gigabit
Attachment Unit Interface(XAUI), Reduced 10 Gigabit Attachment Unit Interface (RXAUI), or
40 Gigabit Attachment Unit Interface (XLAUI) Ethernet interface mode. The device has large
on-chip memory to support full wire speed L2, L3, and L4 filter performance.

Ethernet Infrastructure

ATCA-9405 Installation and Use (6806800M71G)

91

5.2.1 Port Configuration

Table 5-2 lists all 98CX8234 ports by number, the device to which the port is connected, and

the connection type utilized.

Table 5-2 Switch Port Assignment

Ethernet Infrastructure

Port
Group

0 0 XP0 Packet Processor 2, QLM2,

SerDes
Pair Port Destination Interface Type PHY

1

2 XP2 Packet Processor 2, QLM4,

3

4 XRP4 Service Processor 1G, SGMII

5 XRP5 Packet Processor 2 1G, SGMII-RGMII 88E1512

6 XRP6 Packet Processor 2, QLM0,

7 XRP7 10G, RXAUI

8-11 XLG10 Fabric Channel 2 - Data Plane 1000Base-BX (1G)

Data Plane

Data Plane

Data Plane

20G, DXAUI

20G, DXAUI

10G, RXAUI

88X2241
10GBase-KX4 (10G)
10GBase-KR (10G)
10GBase-KR4 (40G)

92

ATCA-9405 Installation and Use (6806800M71G)

Table 5-2 Switch Port Assignment (continued)

Ethernet Infrastructure

Port
Group

1 16 XP16 Packet Processor 1, QLM4,

2 32 XPR32 Serial Redirection 1G, SGMII 88E1512

SerDes
Pair Port Destination Interface Type PHY

20G, DXAUI

17

18 XP18 Packet Processor 1, QLM2,

19

20 XRP20 Packet Processor 1 1G, SGMII-RGMII 88E1512

21 XRP21 Packet Processor 1+2 Front

22 XRP22 Packet Processor 1, QLM0,

23 XRP23 10G, RXAUI

24-27 XLG26 Fabric Channel 1 - Data Plane 1000Base-BX (1G)

33 XRP33 Service Processor 1G, SGMII

34 XRP34 Base Channel 1 - Control Plane 1G, SGMII 88E1322

35 XRP35 Base Channel 2 - Control Plane 1G, SGMII 88E1322

Data Plane

20G, DXAUI

Data Plane

1G, SGMII 88E1512

Ethernet

10G, RXAUI

Data Plane

88X2241
10GBase-KX4 (10G)
10GBase-KR (10G)
10GBase-KR4 (40G)

36 XRP36 RTM Uplink 1G, SGMII SFP

37 XRP37 RTM Uplink 1G, SGMII SFP

38 XP38 Update Channel - Data Plane 20G, DXAUI

39

40-43 XLG42 RTM Uplink, Data Plane, PHY#3 4x RXAUI (10G)

ATCA-9405 Installation and Use (6806800M71G)

NLP10142
1x XLG (40G)

93

Table 5-2 Switch Port Assignment (continued)

Ethernet Infrastructure

Port
Group

3

SerDes
Pair Port Destination Interface Type PHY

48 XRP48 RTM Uplink, Data Plane, PHY#0 10G, RXAUI NLP10142

49 XRP49 10G, RXAUI

50 XRP50 10G, RXAUI

51 XRP51 10G, RXAUI

52 XRP52 RTM Uplink, Data Plane, PHY#1 10G, RXAUI NLP10142

53 XRP53 10G, RXAUI

54 XRP54 10G, RXAUI

55 XRP55 10G, RXAUI

56-59 XLG58 RTM Uplink, Data Plane, PHY#2 4x RXAUI (10G)

5.2.2 Two-Wire Serial Interface

The 98CX8234 includes a two-wire serial interface (TWSI) for each of the four port groups of
the switch. The TWSI supports master/slave transactions and multi master environments
(clock synchronization, interface arbitration). The primary use of the TWSI interface is for
switch initialization after reset, when the TWSI operates as a master for serial ROM initialization
of the switch. On ATCA-9405, the TWSI interface is not used and switch is fully configured via
Switch Management Interface.

NLP10142
1x XLG (40G)

5.2.3 Switch Management Interface

The 98CX8234 provides four PCI Express interfaces for setup, configuration, maintenance, and
management by the external SP. Each PCI Express interface is associated to one of the four port
groups of the switch. The PCI Express interfaces are connected to PEX8608 PCI Express Switch,
as listed in Table 6-1.

94

ATCA-9405 Installation and Use (6806800M71G)

5.2.4 PHY Management Interface

The 98CX8234 includes three separate serial management interfaces (SMI) for management
access to the switch and attached PHY devices. The SMI interface usage and PHY device
connection are listed in Table 5-3.

The Switch contains four, IEEE 802.3 clause 22 compliant, Master SMI Interfaces (MSMI) for
managing external Gigabit Ethernet PHY devices. Each MSMI is associated to one of the four
port groups of the switch. This allows the SP to configure the PHY devices and read back PHY
status information via read/write access through 98CX8234.

Table 5-3 Gigabit Ethernet PHY Management Interface

Used Interface Connected PHY Comment PHY Address

MSMI 0 Not used

MSMI 1 88E1512 Packet Processor 1+2 Front Ethernet 0x00

Ethernet Infrastructure

MSMI 2 88E1512 Serial Redirection 0x00

SFP SFP Module 1 on ARTM 0x01

SFP SFP Module 2 on ARTM 0x02

88E1322 Base Channel 1 0x11

Base Channel 2 0x13

MSMI 3 Not used

The Switch contains four port groups, Master SMI Interfaces (MSMI) and IEEE 802.3 clause 45
compliant for managing external 10 Gigabit Ethernet PHY devices that are connected to the
XAUI ports. Each XSMI is associated to one of the four port groups of the switch.

Table 5-4 10G Ethernet PHY Management Interface

Used Interface Connected PHY Comment PHY Address

MXSMI 0 88X2241 Fabric Channel 2 - Data Plane 0x00

MXSMI 1 88X2241 Fabric Channel 1 - Data Plane 0x00

MXSMI 2 NLP10142 RTM Uplink, Data Plane, PHY#3 0x00 - 0x03

ATCA-9405 Installation and Use (6806800M71G)

95

Table 5-4 10G Ethernet PHY Management Interface (continued)

Used Interface Connected PHY Comment PHY Address

MXSMI 3 NLP10142 RTM Uplink, Data Plane, PHY#0 0x00 - 0x03

NLP10142 RTM Uplink, Data Plane, PHY#1 0x04 – 0x07

NLP10142 RTM Uplink, Data Plane, PHY#2 0x08 – 0x0B

The four Slave SMI Interfaces (CPU_SMI) for host access to all address mapped entities in the
Switch are not used on ATCA-9405. The PCI Express interface is used instead.

5.3 Base Interface

The redundant (dual) ATCA base interface is provided by the Ethernet Switch. The physical
1000Base-T interface (according to PICMG3.0) is implemented using one Marvell 88E1322
Integrated 10/100/1000 Dual Gigabit Ethernet Transceiver.

Ethernet Infrastructure

Two SGMII lanes of the Ethernet Switch are connected to the PHY device. Basic Ethernet PHY
configuration after power-up is done using hardware strapping options. After power-up the
PHY is managed by the Switch via MII interface.

5.4 Fabric Interface

The redundant (dual) ATCA fabric interface is provided by the Ethernet Switch. The physical
interface is implemented using two Marvell 88X2241 10Gbps Quad Channel Transceivers.

The two 88X2241 Transceivers provide eight ports (4 ports each) that are directly connected
to Zone 2 connector P23 row 4+3 (Fabric Channel 1, Port 0-3) and connector P23 row 2+1
(Fabric Channel 2, Port 0-3).

The 88X2241 devices are connected to the Ethernet Switch using ports XLG10 and XLG26. For
more information, refer Ethernet Switch on page 91.

Configuration of the 88X2241 device is done via its MDIO/MDC interface which is connected
to the Ethernet Switch. Fabric Channel 1 Transceiver is connected to MXSMI1 (XLG26)
interface. Fabric Channel 2 Transceiver is connected to MXSMI0 (XLG10) interface.

96

ATCA-9405 Installation and Use (6806800M71G)

Ethernet Infrastructure

The following Ethernet interconnection options are supported by ATCA-9405.

Figure 5-2 Fabric Interface Mode 1x 1000Base-KX (PICMG3.1 Option 1)

Figure 5-3 Fabric Interface Mode 1x 10GBase-KX4 (PICMG3.1 Option 9)

ATCA-9405 Installation and Use (6806800M71G)

97

Figure 5-4 Fabric Interface Mode 4x 10GBase-KR

Ethernet Infrastructure

98

Figure 5-5 Fabric Interface Mode 1x 40GBase-KR4

ATCA-9405 Installation and Use (6806800M71G)

5.5 Update Channel

The Update Channel Interface is provided by the Ethernet Switch and is physically implemented
using one DXAUI port. The DXAUI port is directly connected to Zone 2 connector P20 row 4
(Update Channel Port 0-1) and connector P20 row 3 (Update Channel Port 2-3). E-Keying for
the Update Channel is guaranteed via 98CX8234 Ethernet Switch, which disables its ports after
power-up reset and enables individual ports only by configuration.

Figure 5-6 Update Channel

Ethernet Infrastructure

5.6 Serial Redirection

Terminal Server is implemented to provide serial redirection into base network for serial
consoles of SP and both Packet Processors. The Terminal Server functionality is implemented
using routing logic implemented in the Glue Logic FPGA plus an additional Microcontroller.

The Microcontroller converts the incoming UART ports from Glue Logic FPGA into an Ethernet
interface, which is connected to the Ethernet Switch. For the connection between
Microcontroller and Switch, Marvell 88E1512, Gigabit Ethernet Transceiver in 100Base-T mode
is used.

ATCA-9405 Installation and Use (6806800M71G)

99

Ethernet Infrastructure

The Ethernet Switch is configured at power up via serial EEPROM (refer Two-Wire Serial Interface

on page 94) and therefore supports immediate serial redirection after power up.

Figure 5-7 Serial Redirection

100

ATCA-9405 Installation and Use (6806800M71G)