Znyx Networks bh5700 User Manual
HP bh5700 ATCA 14-Slot Blade Server
Ethernet Switch Blade
First Edition
Manufacturing Part Number: AD171-9603A
June 2006
Ethernet Switch Blade User's Guide release 3.2.2j page ii
Legal Notices
The information in this document is subject to change without notice.
Hewlett-Packard makes no warranty of any kind with regard to this manual, including, but
not limited to, the implied warranties of merchantability and fitness for a particular
purpose. Hewlett- Packard shall not be held liable for errors contained herein or direct,
indirect, special, incidental or consequential damages in connection with the furnishing,
performance, or use of this material.
Restricted Rights Legend. Use, duplication or disclosure by the U.S. Government is subject
to restrictions as set forth in subparagraph (c) (1) (ii) of the Rights in Technical Data and
Computer Software clause at DFARS 252.227-7013 for DOD agencies, and subparagraphs
(c) (1) and (c) (2) of the Commercial Computer Software Restricted Rights clause at FAR
52.227-19 for other agencies.
Information in this document is provided in connection with Intel® products. No
license, express or implied, by estoppel or otherwise, to any intellectual property rights
is granted by this document. Except as provided in Intel’s Terms and Conditions of
Sale for such products, Intel assumes no liability whatsoever, and Intel disclaims any
express or implied warranty, relating to sale and/or use of Intel products including
liability or warranties relating to fitness for a particular purpose, merchantability, or
infringement of any patent, copyright or other intellectual property right. Intel
products are not intended for use in medical, life saving, or life sustaining applications.
Intel may make changes to specifications and product descriptions at any time, without
notice. HEWLETT-PACKARD COMPANY 3000 Hanover Street Palo Alto, California 94304
U.S.A.
Copyright Notice. Copyright ©2003 Hewlett-Packard Development Company, L.P.
Reproduction, adaptation, or translation of this document without prior written
permission is prohibited, except as allowed under the copyright laws.
Additional Copyright Notices. AdvancedTCA® is a registered trademark of the PCI
Industrial Computer Manufacturers Group. Linux® is a registered trademark of Linus
Torvalds. ZNYX Networks, RAIN, RAINlink, OpenArchitect®, CarrierClass and HotSwap
are trademarks or registered trademarks of ZNYX Networks in the United States and/or
other countries. All other marks, trademarks or service marks are the property of their
respective owners.
Ethernet Switch Blade User's Guide release 3.2.2j page iii
About the Ethernet Switch Blade Manual
This manual includes everything you need to begin using the HP Ethernet Switch Blade with
OpenArchitect software, Release 3.2.2j.
Ethernet Switch Blade User's Guide release 3.2.2j page iv
Table of Contents
Chapter 1 Overview of the Ethernet Switch Blade ...........................................................17
High Performance Embedded Switching...................................................................... 17
Advanced TCA® Compliant.........................................................................................17
OpenArchitect Switch Management............................................................................. 18
Extensible Customization of Routing Policies..............................................................18
Powerful CarrierClass Features.....................................................................................18
Ethernet Port Layout..................................................................................................... 18
Ethernet Switch Blade Port Configuration..................................................................19
Base switch Quick Reference.............................................................................. 19
Fabric Switch Quick Reference........................................................................... 19
OpenArchitect Switch Environment............................................................................. 20
OpenArchitect Software Structure................................................................................ 20
Chapter 2 Port Cabling and LED Indicators...................................................................... 23
Connecting the Cables...................................................................................................23
Console Port Cabling................................................................................................23
Connecting to the Console Port................................................................................23
Out of Band Ports (OOB Ports).................................................................................... 24
LED Reference......................................................................................................... 24
Chapter 3 High Availability Networking...........................................................................27
Surviving Partner.......................................................................................................... 27
VRRP........................................................................................................................28
zlmd.......................................................................................................................... 28
Switch Replacement and Reconfiguration............................................................... 29
zspconfig...................................................................................................................29
Example HA Switch Configuration..........................................................................30
Modifying zsp.conf on the Base switch...............................................................31
Modifying zsp_vlan.conf on the Fabric Switch...................................................35
Configuring Surviving Partner......................................................................................42
Central Authority.......................................................................................................... 43
Chapter 4 Fabric Switch Configuration ............................................................................ 46
Two switches, two consoles..........................................................................................46
Connecting to the Fabric Switch Console.....................................................................46
OpenArchitect Configuration Procedure.......................................................................46
Changing the Shell Prompt........................................................................................... 47
Default Configuration Scripts...................................................................................47
Example Configuration Scripts................................................................................ 47
Overview of OpenArchitect VLAN Interfaces.........................................................48
Tagging and Untagging VLANs..........................................................................48
Switch Port Interfaces..........................................................................................49
Layer 2 Switch Configuration.......................................................................................49
Using the S50layer2 Script.................................................................................. 50
Ethernet Switch Blade User's Guide release 3.2.2j page v
Rapid Spanning Tree................................................................................................ 50
To Enable Rapid Spanning Tree:.........................................................................51
Port Path Cost...................................................................................................... 51
Layer 3 Switch Configuration............................................................................. 52
Using the S50layer3 Script.................................................................................. 52
Layer 3 Routing Protocols with GateD ........................................................................54
Using the S55gatedRip1 Script................................................................................ 54
To Modify the GateD Scripts: ................................................................................. 56
Class of Service (COS) ................................................................................................ 57
Egress Queues.......................................................................................................... 57
Ingress Classification................................................................................................57
Marking and Re-marking......................................................................................... 58
Scheduling................................................................................................................ 58
ztmd Explained.................................................................................................... 58
zfilterd Explained..................................................................................................... 58
Running zfilterd........................................................................................................58
Restrictions on Implementation................................................................................59
Conflict Resolution..............................................................................................59
iptables and filtering............................................................................................ 60
Introduction..........................................................................................................60
Packet Walk......................................................................................................... 61
Filter Rules Specifications...................................................................................62
Specifying Source and Destination IP Addresses.................................................... 62
Specifying Protocol............................................................................................. 62
Specifying an ICMP Message Type.................................................................... 62
Specifying TCP or UDP ports............................................................................. 63
Specifying TCP flags...........................................................................................63
Specifying an Interface........................................................................................ 63
Filter Rule Targets............................................................................................... 63
Supported Targets................................................................................................63
Classical Targets..................................................................................................63
ZNYX Targets..................................................................................................... 63
ZACTION Examples........................................................................................... 64
Extensions to the default matches........................................................................64
tc and zqosd ............................................................................................................. 65
FIFO Queues (pfifo and bfifo disciplines)...........................................................65
PRIO and WRR queues....................................................................................... 67
The U32 Filter.......................................................................................................... 69
Combining Queuing Disciplines.............................................................................. 69
Handle Semantics................................................................................................ 70
COPS: Common Open Policy Service..........................................................................70
Protocol Architecture................................................................................................71
OpenArchitect PEP...................................................................................................71
Using pepd................................................................................................................72
Ethernet Switch Blade User's Guide release 3.2.2j page vi
Chapter 5 Fabric Switch Administration........................................................................... 73
Setting the Root Password............................................................................................ 73
Adding Additional Users...............................................................................................73
Setting up a Default Route............................................................................................ 74
Name Service Resolution..............................................................................................74
DHCP Client Configuration..........................................................................................74
DHCP Server Configuration......................................................................................... 74
Network Time Protocol (NTP) Client Configuration................................................... 75
Network File System (NFS) Client Configuration........................................................75
NFS Server Configuration.............................................................................................76
Connecting to the Switch Using FTP............................................................................77
ftpd Server Configuration............................................................................................. 77
Connecting to the Switch Using TFTP......................................................................... 77
TFTPD Server Configuration........................................................................................77
SNMP Agent................................................................................................................. 78
Supported MIBS.......................................................................................................78
Supported Traps........................................................................................................79
SNMP and OpenArchitect Interface Definitions......................................................80
ifStackTable Entries.............................................................................................81
SNMP Configuration................................................................................................81
SNMP Applications..................................................................................................82
Port Mirroring............................................................................................................... 82
Link and LED Control.................................................................................................. 83
Link Event Monitoring..................................................................................................83
Chapter 6 Fabric Switch Maintenance...............................................................................84
Overview of the OpenArchitect switch boot process....................................................84
Saving Changes.............................................................................................................86
Modifying Files and Updating the Switch.................................................................... 86
Recovering from a System Failure................................................................................86
System Boots with a Console Cable............................................................................. 86
Booting with the –i option.............................................................................................87
System Hangs During Boot...........................................................................................88
Booting the Duplicate Flash Image...............................................................................88
Upgrading the OpenArchitect Image............................................................................ 88
Upgrading or Adding Files............................................................................................89
Excluding Saving Files to Flash............................................................................... 89
Upgrading the Switch Driver........................................................................................ 89
Using apt-get................................................................................................................. 90
Chapter 7 Base Switch Configuration................................................................................91
Two switches, two consoles..........................................................................................91
Connecting to the Base Switch Console....................................................................... 91
OpenArchitect Configuration Procedure..................................................................91
Changing the Shell Prompt.......................................................................................92
Default Configuration Scripts..............................................................................92
Ethernet Switch Blade User's Guide release 3.2.2j page vii
Example Configuration Scripts............................................................................92
Overview of OpenArchitect VLAN Interfaces....................................................93
Tagging and Untagging VLANs..........................................................................94
Switch Port Interfaces..........................................................................................94
Layer 2 Switch Configuration.................................................................................. 94
Using the S50layer2 Script.................................................................................. 96
Rapid Spanning Tree................................................................................................ 96
To Enable Rapid Spanning Tree:.........................................................................96
Port Path Cost...................................................................................................... 97
Layer 3 Switch Configuration.................................................................................. 97
Using the S50layer3 Script.................................................................................. 98
Layer 3 Switch Using Multiple VLANs............................................................100
Using the S50multivlan Script...........................................................................100
To Modify the Layer 3 Multivlan Script ......................................................... 102
Modify the example script you copied into the /etc/rcZ.d directory. Adjust and
assign the number of IP addresses as applicable. In the example below, the IP
address is changed for the interface in the ifconfig command line of the script.
........................................................................................................................... 102
Layer 3 Routing Protocols with GateD ................................................................. 102
Using the Provided S55gatedRip1 Script.......................................................... 102
To Modify the GateD Scripts: .......................................................................... 104
Class of Service (COS) ..........................................................................................105
Egress Queues....................................................................................................105
Ingress Classification.........................................................................................105
Marking and Re-marking...................................................................................106
Scheduling......................................................................................................... 106
zcos......................................................................................................................... 106
zfilterd.....................................................................................................................106
ztmd................................................................................................................... 106
Running zfilterd................................................................................................. 107
Restrictions on Implementation.........................................................................107
Conflict Resolution............................................................................................107
iptables and filtering............................................................................................... 108
Introduction........................................................................................................109
Packet Walk....................................................................................................... 110
Filter Rules Specifications.................................................................................110
Specifying Source and Destination IP Addresses..............................................110
Specifying Protocol........................................................................................... 110
Specifying an ICMP Message Type.................................................................. 110
Specifying TCP or UDP ports........................................................................... 111
Specifying TCP flags.........................................................................................111
Specifying an Interface...................................................................................... 111
Filter Rule Targets............................................................................................. 111
Supported Targets..............................................................................................111
Ethernet Switch Blade User's Guide release 3.2.2j page viii
Classical Targets................................................................................................111
ZNYX Targets................................................................................................... 112
ZACTION Examples......................................................................................... 112
Extensions to the default matches......................................................................113
tc: Traffic Control..................................................................................................113
Strict Priority Qdisc................................................................................................113
Weighted Round Robin Qdisc................................................................................114
FIFO Queues (pfifo and bfifo disciplines).........................................................114
Fifo Qdiscs..............................................................................................................115
Using Filters to Direct Packets to a COS Queue....................................................115
Protocol ip.............................................................................................................. 115
Protocol arp............................................................................................................ 116
Protocol all..............................................................................................................116
Matching Specific Ingress Ports.............................................................................116
Advanced Filtering – Policing................................................................................117
Examples............................................................................................................118
Policing Actions..................................................................................................... 118
u32 match selectors used in filters.........................................................................119
zqosd.......................................................................................................................120
PRIO and WRR queues..................................................................................... 121
The U32 Filter........................................................................................................ 123
Combining Queuing Disciplines............................................................................ 124
Handle Semantics.............................................................................................. 124
COPS: Common Open Policy Service................................................................... 124
Protocol Architecture.........................................................................................125
OpenArchitect PEP............................................................................................126
Using pepd......................................................................................................... 126
Chapter 8 Base Switch Administration............................................................................128
Setting the Root Password......................................................................................128
Adding Additional Users........................................................................................128
Setting up a Default Route..................................................................................... 129
Name Service Resolution....................................................................................... 129
DHCP Client Configuration................................................................................... 129
DHCP Server Configuration...................................................................................129
Network Time Protocol (NTP) Client Configuration.............................................130
Network File System (NFS) Client Configuration.................................................130
NFS Server Configuration......................................................................................131
Connecting to the Switch Using FTP..................................................................... 131
ftpd Server Configuration.......................................................................................132
Connecting to the Switch Using TFTP...................................................................132
TFTPD Server Configuration................................................................................. 132
SNMP Agent.......................................................................................................... 132
Supported MIBS................................................................................................ 132
Supported Traps.................................................................................................134
Ethernet Switch Blade User's Guide release 3.2.2j page ix
SNMP and OpenArchitect Interface Definitions............................................... 134
ifStackTable Entries...........................................................................................135
SNMP Configuration......................................................................................... 135
SNMP Applications........................................................................................... 136
Port Mirroring.........................................................................................................136
Link and LED Control............................................................................................137
Link Event Monitoring........................................................................................... 137
Chapter 9 Base Switch Maintenance............................................................................... 138
Overview of the OpenArchitect switch boot process............................................. 138
Saving Changes...................................................................................................... 140
Modifying Files and Updating the Switch..............................................................140
Recovering from a System Failure......................................................................... 140
System Boots with a Console Cable.......................................................................140
Booting with the –i option......................................................................................141
System Hangs During Boot.................................................................................... 142
Booting the Duplicate Flash Image........................................................................ 142
Upgrading the OpenArchitect Image......................................................................142
Upgrading or Adding Files.....................................................................................143
Excluding Saving Files to Flash............................................................................. 143
Upgrading the Switch Driver..................................................................................143
Using apt-get.......................................................................................................... 144
Chapter 10 Connecting to the Ethernet Switch Blade..................................................... 145
Base Interface Hub System:........................................................................................ 145
Ethernet Interfaces: ................................................................................................145
Management Interfaces: ........................................................................................ 145
Fabric Interface Hub System: .....................................................................................146
Ethernet Interfaces: ................................................................................................146
Management Interfaces: ........................................................................................ 146
Connecting to the Base Interface................................................................................ 146
Base Interface Serial Port Connection....................................................................146
Base Interface Out-of-Band Ethernet Connection .................................................147
Connecting to the Fabric Interface .............................................................................148
Fabric Interface Serial Port Connection ................................................................ 148
Fabric Interface Out of Band Ethernet Connection ...............................................149
Chapter 11 Diagnosing a Failed Ethernet Switch Blade Activation ..............................150
Accessing the ShMM.................................................................................................. 152
Verifying Communications Between the ShMM and Switch................................ 152
Critical Threshold Error Reported..................................................................... 152
Analyzing Mstate information for the switch............................................................. 153
Checking the ekey Status From the Shelf Manager.................................................... 153
Chapter 12 Troubleshooting a Failed OpenArchitect Load.............................................155
Recovering from a System Failure .............................................................................157
Booting Without the Overlay File...............................................................................158
Ethernet Switch Blade User's Guide release 3.2.2j page x
Booting the Duplicate Flash Image ............................................................................159
Chapter 13 Network Configuration Problems ............................................................... 160
Interface Overview......................................................................................................160
Physical Interfaces..................................................................................................160
Default Base Interface Configuration.....................................................................161
24 port, Layer 2 Switching, single VLAN.........................................................161
Default Fabric Interface Configuration.................................................................. 163
Editing the S50layer2 script can change the Ethernet Switch Blade Fabric Interface
default configuration. The S50Layer2 script and included example scripts
(/etc/rcZ.d/examples) can be used as templates to create custom scripts. The default
S50layer2 script configures the switch accordingly:..............................................163
Configuration Troubleshooting...................................................................................165
Determining ekey status for a specific slot................................................................. 165
Querying Base Interface ekey Status......................................................................167
Querying Fabric Interface ekey Status................................................................... 168
Network Connectivity Troubleshooting......................................................................170
No Connection........................................................................................................170
Diminished Network Throughput...........................................................................170
Connecting to Devices with Fixed Port Speeds ......................................................... 170
External Fault LED..................................................................................................... 170
Network Tests............................................................................................................. 171
Ping Test ................................................................................................................171
Traceroute Test.......................................................................................................172
Chapter 14 Isolating Hardware Failures.......................................................................... 173
Hardware Subsystem...................................................................................................176
Testing the FlashROMs...............................................................................................177
Testing the Switch Fabric............................................................................................178
Link Status for a single port................................................................................... 178
Link Status for a range of ports.............................................................................. 178
Testing the onboard RAM...........................................................................................179
Testing the Control Processor..................................................................................... 180
Hardware Fault....................................................................................................... 181
Software Error................................................................................................... 181
Chapter 15 High Availability Troubleshooting............................................................... 183
Spontaneous Failover Activity....................................................................................183
Unexpected Fail-back Activity...............................................................................183
Chapter 16 Switch Firmware Overview.......................................................................... 184
Checking the switch firmware version........................................................................184
3.1 Fabric Interface............................................................................................185
Updating the Switch Firmware................................................................................... 186
BootLoader Firmware Upgrade:.............................................................................186
OpenArchitect Firmware Upgrade:........................................................................ 186
IPMC Firmware Upgrade:......................................................................................187
Ethernet Switch Blade User's Guide release 3.2.2j page xi
Chapter 17 Restoring the Factory Default Configuration................................................188
Chapter 18 Before Calling Support..................................................................................189
Appendix A Fabric Switch Command Man Pages........................................................ 191
vrrpconfig ...................................................................................................................192
vrrpd ........................................................................................................................... 194
zbootcfg ......................................................................................................................197
zconfig ........................................................................................................................199
zcos .............................................................................................................................207
zdog ............................................................................................................................211
zfilterd ........................................................................................................................ 213
zflash........................................................................................................................... 214
zl2, zl2mc, zl3host, zl3net, zvlan................................................................................ 216
zgvrpd .........................................................................................................................219
zl2d .............................................................................................................................221
zl3d .............................................................................................................................223
zlc ............................................................................................................................... 225
zlmd ............................................................................................................................228
zlogrotate ....................................................................................................................230
zmirror ........................................................................................................................231
zmnt.............................................................................................................................233
zpeer ........................................................................................................................... 235
zqosd .......................................................................................................................... 238
zrc ...............................................................................................................................240
zreg..............................................................................................................................241
zrld ............................................................................................................................. 243
zsnoopd ...................................................................................................................... 244
zspconfig .................................................................................................................... 246
zstack ..........................................................................................................................253
ztats............................................................................................................................. 258
zsync............................................................................................................................259
ztmd ............................................................................................................................261
brctl(8) ........................................................................................................................263
Appendix B Base Switch Command Man Pages...........................................................266
vrrpconfig ...................................................................................................................267
vrrpd ........................................................................................................................... 269
zbootcfg ......................................................................................................................272
zconfig ........................................................................................................................274
zcos .............................................................................................................................282
zdog ............................................................................................................................286
zffpcounter ................................................................................................................. 288
zfilterd......................................................................................................................... 292
zflash........................................................................................................................... 293
zgmrpd........................................................................................................................ 295
Ethernet Switch Blade User's Guide release 3.2.2j page xii
zgr................................................................................................................................297
zgvrpd..........................................................................................................................300
zl2d..............................................................................................................................302
zl3d..............................................................................................................................304
zlc ............................................................................................................................... 306
zlmd ............................................................................................................................308
zlogrotate ....................................................................................................................310
zmirror ........................................................................................................................311
zmnt.............................................................................................................................314
zpeer ........................................................................................................................... 316
zqosd........................................................................................................................... 319
zrc ...............................................................................................................................321
zreg..............................................................................................................................322
zrld ............................................................................................................................. 324
zsnoopd ...................................................................................................................... 325
zspconfig .................................................................................................................... 328
zstack ..........................................................................................................................336
ztats............................................................................................................................. 340
zsync............................................................................................................................341
ztmd.............................................................................................................................343
brctl(8).........................................................................................................................345
Appendix C Intelligent Platform Management Interface ..............................................348
ISwitch-ShMC Interaction.......................................................................................... 348
Peripheral Management Controller Functional Support............................................. 349
Sensor Reading Example........................................................................................350
Structure of Standard IPMI Commands: From BMC to PMC....................................352
Structure of Standard IPMI Responses: From PMC to BMC..................................... 352
Event Generator ........................................................................................................ 353
IPMB Event message format............................................................................. 353
IPMI Event Message Definitions.......................................................................353
Field Replaceable Unit Inventory Device.............................................................. 353
IPMB Override/Local Status - Event Data 3 for the IPMB link........................354
Table of Figures
Figure 1.1: Fabric Switch Elements...................................................................................20
Figure 1.2: OpenArchitect Software Structure.................................................................. 22
Figure 2.1: LED Reference................................................................................................ 25
Figure 3.1: Host HA Architecture......................................................................................27
Figure 4.1: Fabric VLANs................................................................................................. 48
Figure 4.2: Firewall Flow ................................................................................................ 61
Figure 4.3: COPS Network Architecture........................................................................... 70
Figure 6.1: ROM Devices in Open Architect.................................................................... 84
Figure 6.2: Boot Flow Chart.............................................................................................. 85
Ethernet Switch Blade User's Guide release 3.2.2j page xiii
Figure 6.3: Init Script Flow................................................................................................86
Figure 7.1: Multiple VLANs..............................................................................................94
Figure 7.2: Layer 2 Switch ................................................................................................95
Figure 7.3: Layer 3 Switch ................................................................................................99
Figure 7.4: Multiple VLAN Configuration......................................................................101
Figure 7.5: Firewall Flow ............................................................................................... 109
Figure 7.6: COPS Network Architecture........................................................................ 125
Figure 9.1: ROM Devices in OpenArchitect................................................................... 138
Figure 9.2: Booting up Process Flow..............................................................................139
Figure 9.3: Init Script Flow..............................................................................................140
Figure 10.1: Fabric and Base .......................................................................................... 145
Figure 10.2: Base Interface Serial Port............................................................................ 147
Figure 10.3: Fabric Interface Serial Ports........................................................................ 148
Figure 11.1: Ethernet Switch Blade Activation States.....................................................150
Figure 12.1: OpenArchitect Boot Process....................................................................... 156
Figure 12.2: ROM Devices in OpenArchitect................................................................. 157
Figure 18.1: ROM Devices in OpenArchitect................................................................. 190
Index of Tables
Table 5.1: Supported MIBs................................................................................................79
Table 5.2: Supported Traps................................................................................................80
Table 5.3: Link and SNMP Status..................................................................................... 81
Table 7.1: Port Path Cost................................................................................................... 97
Table 7.2: Policing Actions..............................................................................................119
Table 7.3: U Match Selectors...........................................................................................120
Table 8.1: Supported MIBs..............................................................................................134
Table 8.2: Supported Traps..............................................................................................134
Table 8.3: Physical Link Status on Base Switch..............................................................135
Table 11.1: Troubleshooting States................................................................................. 152
Table 13.1: Ethernet Switch Blade Backplane Interfaces (zre Ports).............................. 160
Table 13.2: Additional Interfaces.................................................................................... 161
Table C.1.: IPMI M States............................................................................................... 349
Table C.2: PMC Controller Support................................................................................ 349
Table C.3: GetSensorReading..........................................................................................350
Table C.4: GetSensorResonse..........................................................................................351
Table C.5: Standard IPMI Commands.............................................................................352
Table C.6: Standard IPMI Responses.............................................................................. 352
Table C.7: Event Message Format...................................................................................353
Table C.8: SEEPROM Space...........................................................................................354
Table C.9.: IPMB Override Status Data.......................................................................... 355
Ethernet Switch Blade User's Guide release 3.2.2j page xiv
Ethernet Switch Blade User's Guide release 3.2.2j page 15
Ethernet Switch Blade User's Guide release 3.2.2j page 16
Chapter 1 Overview of the Ethernet Switch Blade
The Ethernet Switch Blade is a 72-port AdvancedTCA® Hub and providing Gigabit Ethernet.
Up to 14 ATCA node boards may be addressed via the PICMG 3.0 Base Interface and via the
ATCA PICMG 3.1 fabric . The Base and Fabric switching domains are kept totally separate, both
on the physical layer and the software layer. The Ethernet Switch Blade provides a tightly
integrated modular switching platform that enables high-density solutions.
The Ethernet Switch Blade is actually two separate switches, one for the Base ports and one for
the fabric ports. There are two OpenArchitect® operating system images, one for each switch,
allowing the maximum in separation between the control signaling and the data. The modular
design provides great flexibility and control.
Ethernet Switch Blades can support a 10 Gigabit Ethernet Inter-Switch Link (ISL) for the Fabric
Interfaces, and a Gigabit Ethernet ISL for the Base Interface switches. Depending on the version
of OpenArchitect used, the ISL for the Fabric Interface switches may be operated at 10 Gigabits
per second and provide stacking features.
Linux-based OpenArchitect 3 runs on the embedded processors, providing a comprehensive
package for the management of Layer 2 and Layer 3 packet switching. VLAN management and
Layer 2-7 packet classification are also included with a user-friendly interface. OpenArchitect can
be used with a variety of IP routing protocols.
As part of Advanced TCA, the switch incorporates the PICMG 3.0 Intelligent Platform
Management Interface (IPMI) standard for Field Replaceable Unit FRU) management by the
Shelf Manager.
High Performance Embedded Switching
The Ethernet Switch Blade with OpenArchitect combines the performance of silicon-based
switching fabric with flexibility of software-managed routing policies. It provides Base fabric
PICMC 3.0 (1 Gigabit Ethernet ) links to each of the payload slots, plus two to four PICMC 3.1
in-band GigE ports to each node card, and GigE links to management ports and the second
switch. The Ethernet Switch Blade maintains the forwarding table on silicon, providing the
capability to switch and route at full line rate performance on every port.
Advanced TCA® Compliant
The Advanced TCA® standard developed by the PCI Industrial Computer Manufacturer Group
defines an embedded Ethernet environment for high availability chassis. This environment
includes two switch fabric slots that create a dual star Ethernet network to the 14 Base node slots.
Placing the Ethernet Switch Blade in a hub slot provides embedded Ethernet services to each
node card of the chassis. A standard HA configuration is one Ethernet Switch Blade placed in
each of the two hub slots in a chassis for creation of a redundant, high availability system.
Ethernet Switch Blade User's Guide release 3.2.2j page 17
OpenArchitect Switch Management
The OpenArchitect software component – open source Linux, IP protocol stack, control
applications and the OA Engine – runs on two embedded PowerPC microprocessors.
OpenArchitect provides extensive managed IP routing protocols and other open standards for
switch management. Examples include network services; Virtual Redundant Router Protocol;
Routing Information Protocol; Open Shortest Path First; Border Gateway Protocol; Quality of
Service and Class of Service; access control lists; Simple Network Management Protocol MIBs,
Common Open Policy Services and web.
Extensible Customization of Routing Policies
The OpenArchitect software environment enables rapid porting of other UNIX/Linux-based
protocols, including open source software conforming to RFCs and other standards. It also
enables the development of application-specific protocol configuration scripts.
Powerful CarrierClass Features
The Ethernet Switch Blade has High Availability hardware features for advanced
telecommunication applications. The switch implements the PICMG 3.0 Full Hotswap support.
This feature provides field replaceable capabilities so a switch can fail and be replaced without
impacting the operational performance of a chassis.
The PICMG 3.0 Intelligent Platform Management Interface (IPMI) standard is also supported.
IPMI uses message-based interfaces that monitor the physical health characteristics of the
Ethernet Switch Blade. The switch provides operational status information to an IPMI
management application. End customers benefit with advanced notice of potential problems.
The Ethernet Switch Blade also implements the Media Dependent Interface called Auto MDI-X.
Auto MDI-X allows connections to any device, switches, hubs, or systems using a regular
straight-through or crossover Cat 5 cable. The RJ-45 port will auto detect and switch MDI/MDIX modes. This IEEE standard makes cabling – especially between switches – faster and less error
prone.
E-Keying is supported by the Ethernet Switch Blade.
Ethernet Port Layout
The Ethernet Switch Blade has a total of 72 switched Gigabit Ethernet ports. The base fabric is
connected via 24 Gigabit Ethernet ports and the data fabric is connected via 48 Gigabit Ethernet
ports. The Ethernet Switch Blade is actually composed of two separate switches, one for Base
port activity and another for fabric port activity. The Base ports ( control and signaling) are
switched on the Base switch, and the fabric ports ( data ) are switched on the fabric switch, which
provides total separation between system management or control packets, and customer data
packets.
Ethernet Switch Blade User's Guide release 3.2.2j page 18
Ethernet Switch Blade Port Configuration
Base switch Quick Reference
ShelfManager1
ShelfManager2
ISL channel ( Base node2 )
Base nodes 3-14
Base nodes 15,16
Front panel
Fabric Switch Quick Reference
slot zre numbers
3
4
5
6
7
8
9
zre22
zre13
zre23
zre0-11
zre 20-21
zre12, zre14, zre15
zre0-3
zre4-7
zre8-11
zre12-15
zre16-19
zre24-27
zre28-29
10
11
12
13
14
15
16
Inter-switch Link (ISL)
Front panel
zre30-31
zre32-33
zre34-35
zre36-37
zre38, zre39
zre40-41
zre42-43
zre51
zre20-23
Ethernet Switch Blade User's Guide release 3.2.2j page 19
You will find the Ethernet Switch Blade has a straightforward installation and configuration.
UNIX or Linux system management skills and some understanding of network protocols will be
required. Configure the Ethernet Switch Blades to your networking application before you
begin using the OpenArchitect switch.
OpenArchitect Switch Environment
The key elements of the OpenArchitect environment include two embedded Linux operating
systems, OpenArchitect-specific applications and libraries, plus, an innovative switch hardware
design.
OpenArchitect hardware is in many ways similar to typical switch architectures. The primary
difference in OpenArchitect is that the PCI bus that interfaces with the embedded processor and
the switch fabric is at a higher performance level than a typical switch (see Figure 1.1: Fabric
Switch Elements). The use of PCI creates a pipe of significant bandwidth between the processor
and the switch fabric.
The embedded processors, running Linux and the OpenArchitect processes, control the flow of all
traffic by maintaining the switch forwarding tables. These tables define the flow of the switch
traffic. Because they are on the switching chips, packets proceed at line rate.
OpenArchitect Software Structure
Figure 1.1: Fabric Switch Elements
OpenArchitect is based on an embedded Linux operating system and includes a number of ZNYX
Networks-supplied modules. The key element is the Linux routing table, which is crucial in a
Ethernet Switch Blade User's Guide release 3.2.2j page 20
network-enabled Linux implementation.
The purpose of the routing table is to tell the packet forwarding software where to forward the
data packets. In Linux, the packet-forwarding algorithm is operated in software. Normally, the
routing tables are maintained by operator configuration and the various routing protocols that run
in the application environment of Linux.
OpenArchitect uses an innovative new approach for forwarding packets. It provides embedded
software daemons that replicate ( shadow) the Linux routing tables in the silicon-based
forwarding tables (see Figure 1.1: Fabric Switch Elements). In the OpenArchitect switching
environment, the switching chips do the real-time work in switching network packets. The switch
fabric consults its own forwarding tables for each incoming packet; and either filters or forwards
the packet to any egress port, the embedded CPU, or to any combination. The Linux routing
tables, running in software, are used to update the silicon-based tables. This provides both the
flexibility and control of the Linux software environment and the speed of dedicated switching
silicon.
The OpenArchitect environment includes additional features. For example, installing the
OpenArchitect switch gives you immediate implementation of Linux routing protocols. Also, you
have complete support of routing table updates and a standardized method for configuration.
Finally, you can quickly integrate bug fixes, protocol enhancements and additional protocol
implementations from the Linux community. You can also integrate OpenArchitect into other
Linux applications including VPN software, voice over IP protocols, Quality of Service, and
HTML configuration.
RAIN Management API (RMAPI) is a generic interface for passing control data. The
OpenArchitect libraries are implemented completely above RMAPI. The libraries provide a frontend to RMAPI to simplify application writing. Currently one library is implemented, a general
library called zlxlib. As the OpenArchitect application requirements grow, the existing library
will be expanded and additional libraries will be created.
Ethernet Switch Blade User's Guide release 3.2.2j page 21
Linux Application
Level Software
(routed, gated)
ZNYX RAIN Mgt API RMAPI
OpenArchitect Libraries
zlxlib and ztlib
OpenArchitect Application Level Software
(i.e., zconfig, zl3d, zl2d, zsync)
Linux
Protocol
Stack
Linux
Routing
Tables
Open Architect Driver
PCI Bus
Switch
Fabric
Linux
Application
Environment
Linux
Kernel
Figure 1.2: OpenArchitect Software Structure
OpenArchitect applications are used to program and configure the Ethernet Switch Blade. These
applications are implemented above the libraries and RMAPI.
Ethernet Switch Blade User's Guide release 3.2.2j page 22
Chapter 2 Port Cabling and LED Indicators
The PICMG 3.1 standard defines an embedded Ethernet environment for Telco chassis. This
environment includes two switch fabric slots that create a dual star Ethernet network to the
fourteen node slots. Placing the Ethernet Switch Blade in a hub slot provides embedded Ethernet
services to each node card across the Packet Switching Backplane of the chassis. A standard
configuration is to place a Ethernet Switch Blade in each hub slot creating a redundant, high
availability system. This chapter provides information on the Ethernet Switch Blade port
connectors and LED indicators.
Connecting the Cables
Your switch setup may require some or all of the following types of cables: 10/100/1000 Port
Cabling
Category 5 cabling is required for all external ports. Be sure that your cable length is within the
minimum and maximum length restrictions for the Ethernet, otherwise you could experience
signal or data loss. All copper GigE ports on the Ethernet Switch Blade are auto-MDI sensing
and will automatically determine whether or not an MDI (straight-through) or MDI-X (crossover)
cable is attached.
Console Port Cabling
The switch console can be accessed via one RJ-45 10/100 service port located on the front panel
of the Ethernet Switch Blade.
NOTE: There are two switch portions that make up a Ethernet Switch Blade unit. Each
switch portion, Base and fabric, has its own console ports, and requires its own console
cable or OOB Ethernet cable.
The RS-232 configured RJ-45 connector console port on the front panel can be used to recover
from a system failure. It is used for maintenance only, and is generally not connected. Use a HP
console cable (P/N A6900-63006) provided with the HP bh5700 ATCA 14-Slot Blade Server, in
combination with a Modem Eliminator cable, to access the switch software through the console
port. Refer to the HP bh5700 ATCA 14-Slot Blade Server Installation Guide for additional
information.
Connecting to the Console Port
To attach the console cable to the OpenArchitect Base or fabric switch:
1. Plug the RJ-45 end of the console cable into the RJ-45 Console Port on the front.
2. Connect the Modem Eliminator cable to the DB-9 connector on the console cable.
3. Connect the other end of the Modem Eliminator cable to a standard COM port (9600, n,
8, 1).
Ethernet Switch Blade User's Guide release 3.2.2j page 23
4. Reinsert the switch into the shelf chassis and power up.
Use a terminal emulation program to access the switch console.
Out of Band Ports (OOB Ports)
Each switch, fabric and Base, in a Ethernet Switch Blade unit has out-of-band (OOB) Ethernet
ports on the front panel. This is an alternative maintenance port supplying Ethernet connectivity
instead of serial connectivity and is connected only when performing switch maintenance
activities. Use ifconfig to bring up and configure the OOB ports. The OOB ports are 100 full
duplex, not auto-sensing. The front OOB port is eth0, and the rear (not implemented with this
release) is eth1.
LED Reference
See Figure 2.1 for a schematic view of the front of a typical Ethernet Switch Blade board. Note
that there are out-of-band ports, RS232 ports, a USB port, and 10 Gig egress ports (not
implemented in this release). In-band ports from the Base and fabric switches have LED status
lights controlled from the LED Mode button. Press the button successively to display the Base
switch ports, fabric switch ports 0-23, and finally the fabric switch ports 24-47. There are
separate LEDs for the out-of-band ports, and the ATCA status functions.
Ethernet Switch Blade User's Guide release 3.2.2j page 24
Figure 2.1: LED Reference
Ethernet Switch Blade User's Guide release 3.2.2j page 25
Ethernet Switch Blade User's Guide release 3.2.2j page 26
Chapter 3 High Availability Networking
Architecture
High availability networking is achieved by eliminating any single point of failure through
redundant connectivity: Redundant cables, switches and network interfaces for hardware,
combined with HA software solutions on both the hosts and switches to control the HA hardware
and maintain connectivity. An HA solution called Surviving Partner is provided on the switch.
For host-side HA, the most common solution is to use the Linux bonding driver. HA solutions
like the Linux bonding driver present a single, virtual interface to the protocol stack while
managing multiple physical links. Figure 3.1: Host HA Architecture shows the relation of the
protocol stack, a bonding driver and physical ports.
Figure 3.1: Host HA
A failover between physical links can be made very quickly without requiring change to the IP or
MAC address of the virtual interface, effectively transparent from the applications point of view.
With redundant links from a switch (or switches) to the host, one link is maintained as the
ACTIVE link and the other as STANDBY. If the ACTIVE link were to go down, the STANDBY
becomes the new ACTIVE, while presenting the same virtual interface to the host.
NOTE: It is important that the bonding solution provide an active-backup mode. For the
Linux bonding driver set “mode == 1” see the http://sourceforge.net/projects/bonding/
documentation for more information. Use the recommendations for Linux kernel 2.4x not
2.6x.
Redundant connections provide an ACTIVE and STANDBY link to a switch, or provide
redundant links between more than one switch. In the case of more than one switch, a complete
HA solution requires a switch-based HA solution.
Surviving Partner
Surviving Partner is a switch-based HA solution. Surviving Partner runs on the switches to
provide transition of Layer 2 and Layer 3 switching functionality between two or more switches.
Surviving Partner is comprised of many interactive protocols and processes including VRRP,
zlmd, zlc, and others.
Ethernet Switch Blade User's Guide release 3.2.2j page 27
VRRP
Since most end nodes use default router addresses, the change of the default router address during
a switch failover would require the end nodes to reconfigure. Layer 3 switches that failover must
maintain the default router address to maintain the end node's IP transparent failover. The Virtual
Router Redundancy Protocol (VRRP, RFC 2338) running in the Surviving Partner switches
provides transparent movement of the default router address. VRRP maintains the notion of a
Master switch and one or more Backup switches. This group of switches presents a virtual router
IP address that can be used by hosts on that net as their default route.
If a Backup switch determines the Master switch is no longer available, one of the Backup
switches will assume the role as Master. Physically, each switch maintains a link to the local
network. Only the Master switch answers to the default gateway, and the hosts on that net have
no need to relearn the router address.
In an HA configuration, the goal is to avoid any single point of failure. VRRP provides a good
mechanism to provide a static route for a local network, but a true HA configuration must also
provide redundant connections for the host. Providing a virtual router for the local network is not
enough. Take the simple case of two hosts on the local network with a connection to the virtual
router. Each host needs a connection to each physical switch participating in VRRP. In the
simplest configuration, each host would have one connection to the network. An HA solution
would include redundant connections from each host to each switch in the virtual router.
Combining the features of Surviving Partner on the switches and HA bonding drivers on the hosts
allows implementation of this true HA configuration.
zlmd
In addition to complete switch failover, single link failure must be properly handled. The Link
Monitor Daemon zlmd, monitors the link status of each port. If a link goes down, zlmd
communicates with the VRRP daemon (vrrpd) to change its priority. Changing the VRRP
priority results in movement of switching functionality. By combining zlmd with the zlc
application, links connected to hosts that have not failed can be deterministically moved to the
new master switch if desired. Supported modes include:
• switch - The switch with the greatest number of UP links becomes the Master for all
VLANs under HA management.
• Vlan - The switch with the greatest number of UP links in that particular VLAN becomes
the Master for that particular VLAN. If the switch has additional VLANs, they each
change independently.
• Port - The Master will remain the Master for that particular VLAN until all ports in that
VLAN are down. The Backup then becomes the new Master for that VLAN. Failed links
move their connectivity through the Backup Switch and the switch interconnect to reach
the Master Switch. This option alleviates the need to move all nodes to a new switch just
because a single link goes down.
NOTE: All modes require inclusion of the interconnect in the VLAN. The ISL
connection between the two Base switches is port 23 for the Ethernet Switch Blade. The
ISL connection between the two fabric slots in port 51.
Ethernet Switch Blade User's Guide release 3.2.2j page 28
Switch Replacement and Reconfiguration
When a switch fails, it must be replaced. The replacement switch will likely require proper
configuration. For transparent switch replacement, the newly replaced switch must learn its
configuration from its Surviving Partner.
In a simple failover scenario, Host A and Host B are configured with failover between two host
ports, one port connected to Switch A and the other connected to Switch B. Assume Switch A
provides connectivity between Host A and Host B. If Switch A fails, the active link on each host
moves over to the port connected to Switch B. Surviving Partner software on Switch B
recognizes that Switch A has failed, and assumes the role of switching traffic between Host A and
Host B. When the failed Switch A is replaced with a new Switch A', Switch A' will learn its
network configuration from the surviving partner Switch B. Switch A' is now ready as a backup
to Switch B in case of failure of Switch B.
This is achieved through the use of DHCP. When a switch becomes a VRRP Master, a DHCP
server is started with a pointer to a configuration file that contains configuration information for
its partners. The replacement switch comes up running DHCP client to retrieve its configuration.
Proper configuration of Surviving Partner requires coordinated configuration of many different
processes, including vrrpd, zlmd, zlc, and dhcpd. The daemon processes run scripts to
perform their actions. Because these scripts are complex and inter-dependent, a configuration
application called zspconfig is used to build them.
The basic steps to configuring Surviving Partner are:
1. Determine your desired configuration.
2. Modify the configuration file (
the default
3. Configure startup scripts or other scripts such as gated routing scripts and vrrp
configuration scripts.
4. Run
5.
Run zspconfig –u
zspconfig
zspconfig performs the job of building the scripts based on a provided input file locally, or
from a remote machine. A text-based configuration file provides input to zspconfig. Example
configuration files are included on the switch in /etc/rcZ.d/surviving_partner. The
result of zspconfig is to create several configuration files and runtime shell scripts, and
optionally start the Surviving Partner processes. Scripts are generated for configuring VLANs,
starting the network, and starting the vrrpd and zlmd daemons.
zspconfig can also used by sibling backup switches to retrieve configuration from the
Surviving Partner and start the vrrpd and zlmd daemons. zspconfig is generally only run
once to configure Surviving Partner.
) to use as input to the configuration utility (zspconfig).
zspconfig
on the Master system.
on the Backup/Sibling system(s).
/etc/rcZ.d/surviving_partner/zsp_DC.conf
is
Ethernet Switch Blade User's Guide release 3.2.2j page 29
The configuration and runtime scripts created are as follows:
• S70Surviving_partner Switch initialization script that is run at boot time. This
script will restart the switch with the original configuration given to zspconfig.
Optionally, zspconfig will run this script from the initial invocation.
• zsp.conf.<n> - zspconfig configuration file that contains the configuration of
the sibling backup switches. The <n> is used to distinguish potentially more than one
backup switch. This configuration file is placed in /tftpboot, and is retrieved via
DHCP during configuration of the backup switch by zspconfig with the “-u” option
or, by a replacement switch on boot up.
• vrrpd.conf - Configuration script for the VRRP daemon. This configuration is
used when the S70Surviving_partner script launches vrrpd. There is a line in this file for
each virtual router address vrrpd will manage.
• dhcpd.conf - Configuration script used by dhcpd when the switch becomes
master. dhcpd is also used to give replacement switches their configuration scripts.
Namely a zsp_.conf<n> file that can be input to zspconfig with the -u flag.
• dhclient.conf - If zspconfig is executed with the -u flag, a dhclient.conf file
is created, and then dhclient is used to retrieve a zspconfig configuration file from the
/tftpboot area of the Master switch.
• vrrpd.script - Runtime script that executes each time the vrrpd changes state.
This script starts and stops dhcpd, and toggles down RAINlink ports to force the
RAINlink nodes to a new Master switch.
• zlmd.script - Runtime script executed by zlmd when a link goes up or down. This
script modifies the priority of the vrrpd that in turn may cause the VRRP Master to move
from one sibling switch to another.
After the scripts are created, zspconfig may run the
S70Surviving_partner
script to start
the Surviving Partner tasks. The tasks started are vrrpd, zlmd, and dhcpd.
The vrrpd and zlmd daemons run scripts to perform their actions. When vrrpd changes state
between Master and Backup, it runs a script that starts and stops dhcpd. When zlmd sees a link
go up or down, it runs a script that communicates with vrrpd via vrrpconfig.
Example HA Switch Configuration
The following walks through a basic Surviving Partner configuration typical for an HA setup.
Assume an HA chassis with multiple hosts, such as single-board CPUs, and two switches
configured for Surviving Partner. Each of the hosts has two Base Ethernet ports providing a link
to each of the Base switches and up to four fabric Ethernet ports providing links to each of the
Fabric switches.
Each host runs Linux bonding drivers (or ZNYX OA Node software with embedded RAINlink)
with the ports configured for failover. An interlink provides communication between the Base
switches. Another interlink provides communication between the Fabric switches.
Ethernet Switch Blade User's Guide release 3.2.2j page 30
Loading...