Today most datacenter networks run with switches from a single vendor. Although most of the protocols used are standardized,
there are a number of proprietary ones – especially redundancy and management protocols. Other features may be so
individual that interoperability is possible but not simple.
It is therefore sometimes a challenge to integrate switches from one vendor into a network that has been build using a different
vendor. This paper is intended to guide the reader with the task of integrating BX600 SB9 switches into Cisco networks.
A number of major aspects that are common to most datacenter networks are covered and have been tested in Fujitsu
Siemens’ laboratories. All the features of Cisco switches mentioned in this paper have been tested with Catalyst 3560 and
Catalyst 3750 series switches.
The following Cisco IOS software was used for the integration tests:
Catalyst 3750 IOS 12.2(25)SEE1 Advanced IP Services
Catalyst 3560 IOS 12.2(25)SEE1 Advanced IP Services
The PRIMERGY BX600 GbE switch is an integrated Gigabit Ethernet switch for use in the PRIMERGY BX600 chassis. Up to
four switches can be installed, and each installed switch offers ten 1Gbit downlink ports to the midplane for connection to server
blades. The PRIMERGY GbE switch comes in two variants as regards the external ports: one with six 1 Gbit uplink ports
(RJ45), and one with six 1 Gbit uplink ports and two 10 Gbit uplink ports (XFP, CX4). The two 10 Gbit ports of the second
variant can be connected by means of an XFP module and a CX4 cable. Layer 2/3/4 functionalities are supported.
PRIMERGY BX600 GbE switch variant 1:
PRIMERGY BX600 GbE switch variant 2:
• Six 1 Gbit/s Ethernet RJ45 ports
• Six 1 Gbit/s Ethernet RJ45 ports
• Two 10 Gbit/s Ethernet ports (XFP, CX4)
• Infiniband cable 10m (10GBASE-CX4)
must be ordered separately
•XFP multimode module (10GBASE-SR)
must be ordered separately
White Paper Issue: October 2006 Integration of BX600 SB9 Switches in Cisco Networks Page 4 / 47
2 Switch Connectivity
2.1 Auto Negotiation
2.1.1 Introduction
The SB9 is equipped with at least six Gigabit Ethernet ports which are implemented as specified in the 1000BaseT standard.
(Since ten Gigabit Ethernet is not usual in datacenters’ server access layer, the 10GBaseCX4 and XFP interfaces that are also
available are not covered here.)
These ports can be run with different data rates and different duplex settings comparable to Cisco Switches. Table 1 shows the
possible combinations of a Cisco Switch and an SB9. Only the combinations marked green are viable: the combinations marked
red are risky because they will lead to a duplex failure.
SB9
Auto
Fix Half Duplex 10
Fix Half Duplex 100
Fix Full Duplex 10
Fix Full Duplex 100
Fix Half Duplex 10 Ok N/A BAD N/A Ok
Fix Half Duplex 100 N/A Ok N/A BAD Ok
Fix Full Duplex 10 N/A N/A Ok N/A BAD
Fix Full Duplex 100 N/A BAD N/A Ok BAD
Cisco Switch
Fix Full Duplex 1000 N/A N/A N/A N/A Ok
Auto Ok Ok BAD BAD Ok
Table 1 : Speed and Duplex Settings
During the ports’ autonegotiation phase the flow control mechanism can also be negotiated. Switches are not the best location
for buffering packets during congestion; this mechanism should therefore not be activated on links between switches but
preferably between servers and switches. In this case the server would be able to buffer the packets if the switch were to detect
congestion on the uplink. Since flow control depends very much on the server hardware and software, this issue is not covered
in this paper.
2.1.2 Recommended Solution
We recommend setting the ports on both sides to auto negotiation. In this setting the switches will negotiate their capabilities
and will find the best possible setting. When connected to the usual 1000BaseT port of a Cisco switch using a crossover or
straight thru 8 wire Cat5E, or (better) a patch cable, the SB9 will negotiate 1000 Mbit with full duplex.
Flow control should be disabled between switches.
2.1.3 Configuration
You set a port of the SB9 (e.g. 0/12) to auto negotiation and no flow control by entering the following commands in configuration
mode:
interface 0/12
negotiate
no storm-control flowcontrol
exit
Here is the corresponding Cisco configuration:
interface GigabitEthernet0/2
speed auto
duplex auto
flowcontrol receive off
end
White Paper Issue: October 2006 Integration of BX600 SB9 Switches in Cisco Networks Page 5 / 47
2.2 Port Aggregation
2.2.1 Introduction
You will usually need more than 1 Gbit when connecting an SB9 switch in a datacenter. In this case two or more links are set up
to form a port-channel, also known as a Fast Ethernet Channel (FEC) or Gigabit Ethernet Channel (GEC) in Cisco networks.
Figure 1 shows a typical uplink configuration for an SB9: One port-channel connects to Cisco switch A and a second one
connects to Cisco switch B. Each port-channel is formed of two links running with 1000 Mbit in full duplex mode. The
redundancy mechanisms between these links will be discussed later. In principle, port-channels can be configured statically or
using a port aggregation protocol. Cisco supports LACP as specified in 802.3ad and their proprietary PagP, while the SB9
supports LACP as specified in 802.3ad.
Using static or LACP dynamic configuration, you can form up to six GE links between the SB9 and one other switch.
Figure 1 : Typical uplink configuration for SB9
Table 2 shows the possible combinations of port-channel settings between SB9 and Cisco switches. The combinations marked
red are very risky and would lead to networks loops.
SB9
No Channel
LACP
Static
No channel OK OK !!!
Active OK OK !!!
Passive OK OK !!!
Cisco
On !!! !!! OK
Table 2 : Possible port-channel configurations
So called “split channels”; where one channel from one switch is terminated at two other switches; are supported neither by the
SB9 nor by Cisco switches.
2.2.2 Recommended Solution
Although Cisco switches and SB9 both support LACP, and although this feature has been tested to be compatible between
these devices, we recommend using static configured trunks. This is the best practice to minimize the risk of incompatibilities
and misconfigurations.
Caution: In order to avoid loops in the network, please be sure that the affected ports of a port-channel are shut
down during the configuration process. Generating loops in a datacenter network may cause serious
network problems!
White Paper Issue: October 2006 Integration of BX600 SB9 Switches in Cisco Networks Page 6 / 47
2.2.3 Configuration
The setup in Figure 1 would be configured in the following steps:
• Step 1: Shut down the affected ports to avoid loops
• Step 2: Set up the port-channel
• Step 3: Bring up the affected ports
• Step 4: Verify the operation of the port-channels
Step 1: Shut down the affected ports to avoid loops
! SB9
interface range 0/11 – 0/14
shutdown
exit
! Cisco A
interface range Gi 0/1 – 2
shutdown
end
! Cisco B
interface range Gi 0/1 – 2
shutdown
end
Step 2: Set up the port-channel
! SB9
port-channel Po1
interface 1/1
exit
port-channel Po2
interface 1/2
exit
interface range 0/11 – 0/12
channel-group 1/1
exit
interface 0/13 – 0/14
channel-group 1/2
exit
interface 1/1
! static configuration – no LACP
staticcapability
exit
interface 1/2
! static configuration – no LACP
staticcapability
exit
end
! Cisco A
interface Port-channel1
!
interface range Gi 0/1 - 2
channel-group 1 mode on
end
! Cisco B
interface Port-channel2
!
interface range Gi 0/1 - 2
White Paper Issue: October 2006 Integration of BX600 SB9 Switches in Cisco Networks Page 7 / 47
channel-group 2 mode on
end
Step 3: Bring up the affected ports
! SB9
interface range 0/11 – 0/14
no shutdown
exit
end
! Cisco A
interface Po 1
no shutdown
end
! Cisco B
interface Po 2
no shutdown
end
Step 4: Verify the operation of the port-channels
! SB9
(SB9) #show port-channel
Logical Interface Port-Channel Name Link State Mbr Ports Active Ports
----------------- ----------------- ---------- ---------- -----------1/1 Po1 Up 0/11,0/12 0/11,0/12
1/2 Po2 Up 0/13,0/14 0/13,0/14
! Cisco A
Cisco-A#show etherchannel summary
Flags: D - down P - in port-channel
I - stand-alone s - suspended
H - Hot-standby (LACP only)
R - Layer3 S - Layer2
U - in use f - failed to allocate aggregator
u - unsuitable for bundling
w - waiting to be aggregated
d - default port
Number of channel-groups in use: 1
Number of aggregators: 1
Cisco-A#show etherchannel summary
Flags: D - down P - in port-channel
I - stand-alone s - suspended
H - Hot-standby (LACP only)
R - Layer3 S - Layer2
U - in use f - failed to allocate aggregator
u - unsuitable for bundling
w - waiting to be aggregated
d - default port
Number of channel-groups in use: 1
Number of aggregators: 1
White Paper Issue: October 2006 Integration of BX600 SB9 Switches in Cisco Networks Page 8 / 47
2.3 VLANs and Trunks
2.3.1 Introduction
Most network administrators want to partition their network into multiple broadcast domains to provide better network stability
and better information security. This is implemented using virtual LAN technology (VLANs) which provides multiple virtual LAN
segments in one switched network domain as specified in the standard 802.1Q.
A number of protocols have been developed to simplify the management of such VLANs. While Cisco uses its own proprietary
VLAN Trunking Protocol (VTP), the IEEE describes the GARP VLAN Registration Protocol (GVRP) which has been
implemented in the SB9.
Figure 2 : VLAN Trunk between SB9 and Cisco Switch
When multiple switches are interconnected there is often a need to transport multiple VLANs over one line. This technique is
called VLAN Trunking and is described in the IEEE standard 802.1Q and implemented in the SB9. Some older Cisco switches
implement a proprietary and incompatible ISL, but all devices found in modern datacenters will support 802.1Q trunks. Figure 2
shows a typical setup between a Cisco and an SB9 switch, whereby a port-channel is combined with a VLAN trunk.
It is important to know the role of the so-called native VLAN on an 802.1Q trunk. All the packets on the trunk are encapsulated
in 802.1Q packets, which means that a header containing the VLAN number and certain other information is added to the
packet before it is transported over the trunk. Only the packets of the native VLAN are untagged for a variety of reasons. In most
installations, VLAN1 is configured as native VLAN which is used for a number of protocols, such as VTP, CDP, STP, etc.
2.3.2 Recommended Solution
Cisco’s VTP and standard GVRP are not compatible. Since a VLAN registration protocol is useful only when applied to several
switches within a switch domain, GVRP is not recommended in a Cisco environment.
A number of features of the current version V 2.0 make it neither usual nor advisable to use VTP in datacenter networks:
• The design of the VTP server and client concept is extremely delicate: if you bring in a VTP client switch with a higher
configuration version number than the rest of the network, all the switches will copy the VLAN database from this
switch. This will be a disaster if the new switch has been used in a laboratory and one or more VLANs had been
deleted in the meantime.
• Manual trunk configuration is very deterministic as to which VLAN is on which trunk. This will simplify troubleshooting.
• Manual trunk configuration may help the administrator to set up a simple load sharing.
We therefore recommend using manual VLAN registration in a Cisco datacenter network.
Since the SB9 does not support ISL, the only solution for VLAN trunks to Cisco switches is IEEE 802.1Q. When STP is used,
which is the case for most of datacenters, it is necessary to use a native VLAN because the standard defines that BPDUs have
to be transported untagged. (See also Spanning Tree)
Cisco recommends not using VLAN 1 for anything productive. It therefore makes sense to configure the management IP
address of the SB9 into another VLAN, but it is nevertheless important to have one native VLAN defined on the trunk.
2.3.3 Configuration
You set up a VLAN trunk as shown in Figure 2 and our recommendations by performing the following steps:
• Step 1: Configure the port-channels
• Step 2: Define the VLANs
• Step 3: Configure VLAN trunk
• Step 4: Verify the VLAN trunk
White Paper Issue: October 2006 Integration of BX600 SB9 Switches in Cisco Networks Page 9 / 47
Step 1: Configure the port-channels
Please refer chapter 2.2
Step 2: Define the VLANs
! SB9
! Configure the VLANs (VLAN 1 is default and can’t be configured
vlan database
vlan 10
vlan name 10 VLAN-10
vlan 20
vlan name 20 VLAN-20
exit
! Cisco-A
! Configure the VLANs (VLAN 1 is default and can’t be configured
vlan 10
name VLAN-10
!
vlan 20
name VLAN-20
!
vlan 30
name VLAN-30
Step 3: Configure VLAN trunk
! SB9
! Definition of the port-channel
port-channel Po1
interface 1/1
exit
interface 0/11
channel-group 1/1
exit
interface 0/12
channel-group 1/1
exit
! Configure the interfaces for VLAN trunking
!
interface range 0/11 – 0/12
! the native vlan 1 is default and normally not displayed in configuration
switchport native vlan 1
switchport allowed vlan add 10
switchport tagging 10
switchport allowed vlan add 20
switchport tagging 20
exit
! Configure the port-channel for VLAN trunking
!
interface 1/1
staticcapability
! the native vlan 1 is default and normally not displayed in configuration
switchport native vlan 1
switchport allowed vlan add 10
switchport tagging 10
switchport allowed vlan add 20
switchport tagging 20
exit
Port Vlans allowed and active in management domain
Po1 1,10,20
Port Vlans in spanning tree forwarding state and not pruned
Po1 1,10,20
White Paper Issue: October 2006 Integration of BX600 SB9 Switches in Cisco Networks Page 11 / 47
2.4 Spanning Tree Protocol
2.4.1 Introduction
When the only standard for spanning tree protocols in LANs was STP, as specified in 802.1D, Cisco developed a number of
proprietary protocol enhancements. Some of these were adopted into the RSTP standard but others were not. Cisco therefore
also modified their RSTP implementation to be compatible with their enhanced STP. Table 3 shows all current STP
implementations.
STP 802.1D STP as specified in 802.1D. Slow convergence, does not
support multiple instances for VLAN trunks.
RSTP 802.1w Rapid STP as specified in 802.1w. Fast convergence, does not
support multiple instances for VLAN trunks.
MSTP 802.1s Multiple Instance STP as specified in 802.1s. Fast convergence,
support multiple instances for VLAN trunks
PVST+ STP as specified in 802.1D with the following enhancements:
• port-fast feature
• uplink-fast feature
• backbone-fast features
• spanning tree for each VLAN
Fast convergence, compatible to 802.1D even on VLAN trunks.
PVST Like PVST+ but supporting only ISL trunks Cisco: proprietary solution
RAPID-PVST RSTP as specified in 802.1w with the following enhancements:
• spanning tree for each VLAN
Fast convergence, compatible to 802.1D even on VLAN trunks.
Table 3 : Spanning tree protocol implementations
When connecting switches without VLAN trunks, PVST+ and STP are compatible with RSTP and RAPID-PVST respectively
without any problems. Other combinations are discussed in the following section.
SB9: conforms to the standard
Cisco: supported only on access ports
not on trunks.
SB9: conforms to the standard
Cisco: supported only on access ports
not on trunks.
SB9: conforms to the standard
Cisco: conforms to the standard but not
common in Cisco environments
Cisco: proprietary solution
SB9: not supported yet
Cisco: proprietary solution
SB9: not supported yet
Running ST P 802.1D with PVST+ on VLAN Trunks
When running STP over VLAN trunks, MSTP is the only STP protocol implemented by Cisco that completely complies with the
IEEE standard. This is unfortunately not usually used in datacenter networks, where PVST+ and RAPID-PVST are more
common.
Unlike 802.1D, in which only one STP instance is used to control the STP state of the trunk, PVST+ runs one STP instance per
VLAN, sends BPDUs and maintains one STP state per VLAN on a trunk. In addition to this major deviation from the standard,
Cisco added a number of minor changes, such as the port-fast, uplink-fast and backbone-fast features, which have only local
effects and do not limit their interoperability.
PVST+ is also compatible to STP as specified in 802.1D when there is a native VLAN on the trunk. Figure 3 shows a scenario in
which two Cisco switches are running PVST+ and an SB9 is running STP as specified in 802.1D.
Figure 3 : Combining PVST+ and 802.1D
Switch A is configured as root bridge, while switch B will take over the root role when A fails. Since switch A sends untagged
BPDUs from VLAN 1 to Po1, the SB9 uses Po1 as root port. Po2 of SB9 will take on port role “alternate” and will be in the state
“discarding” and will not send any BPDUs at this port. Switch B will therefore also set its port Po2 to “designated” and
“forwarding”. The SB9 takes all decisions as indicated by the BPDUs in VLAN 1, and all other BPDUs will be ignored. It is
White Paper Issue: October 2006 Integration of BX600 SB9 Switches in Cisco Networks Page 12 / 47
therefore important that one native VLAN is defined at both VLAN trunks. Cisco recommends that this native VLAN should be
the same for both trunks to the SB9.
If the Po1 link or switch A itself fails, the SB9 will change the role of Po2 to “designated” and its state to “forwarding”, after going
through the state “learning”. According to the standard this will lead to a failover time of approximately twice the forward delay,
which in normal cases will be about 30 seconds. Depending of the size of the network this time can be reduced by tuning the
STP timers, but this must be done very carefully in order to provide a stable network. Please refer the standard 802.1D or
Cisco’s recommendations for timer tuning.
When the SB9 is running 802.1D it supports features such as Cisco’s proprietary port-fast when the “spanning-tree edgeport”
command is applied. This means that an access port will take on the state “forwarding” and will omit the states “listening” and
“learning”. This is needed when PXE boot mechanisms are used.
Running PVST+ on VLAN Trunks while disabling STP at the SB9
When STP is disabled at the SB9 it bridges the BDPUs without any modifications. Figure 4 shows this scenario.
Root port
forwarding
Cisco B
priority 4096 for all vlans
Cisco A
priority 0 for all vlans
Designated port
forwarding
Po3Po3
SB9
STP disabled
Designated port
forwarding
Po1
Po1
Po2
Po2
Alternate
discarding
On all trunks:
VLAN 1 native
VLAN 10 tagged
VLAN 20 tagged
Figure 4 : PVST+ while STP is disabled at SB9
Since switch B receives the BPDUs of switch A, its port Po2 will get the role “alternate” and it will take on the state “discarding”.
The SB9 will not be involved in any decisions while the topology is changing.
If the link Po1 fails, switch B will not receive any BPDUs at Po2. After three times the “hello” interval, Po2 will initiate its change
to the role “designated” and will subsequently take on the “forwarding” state. Since no STP is enabled at the SB9, all the
switch’s ports will be enabled and forwarding as soon as they come up.
Without STP timer tuning, worst-case failover times resulting from link or switch failures were found to be approximately 45
seconds.
White Paper Issue: October 2006 Integration of BX600 SB9 Switches in Cisco Networks Page 13 / 47
Rapid Spanning Tree
The standard IEEE 802.1w (RSTP) defines only BPDUs in the native VLAN as implemented by the SB9. Cisco also enhanced
RSTP to RAPID-PVST which is compatible to RSTP in a number of ways. Figure 5 shows this scenario.
Figure 5 : Combining RAPID-PVST and 802.1w
All RSTP features are functioning for the native (in this example VLAN1). Since the SB9 implements the standard, and does not
know about tagged BPDUs, RAPID-PVST has the same restrictions as PVST+.
There is an additional problem due to the fact that RSTP generates a Topology Change Notification (TCN) only when changing
a port to the state “designated”. If the Po1 link in Figure 5 fails, port Po1 of switch A will go down and will not generate a TCN as
specified in 802.1w. SB9 will change the role of port Po2 to root port and its state to “forwarding” and will generate a TCN as
specified in 802.1w on the native VLAN. This has the effect that the Cisco switches will flush their MAC address tables of
VLAN 1 but not for the other VLANs.
White Paper Issue: October 2006 Integration of BX600 SB9 Switches in Cisco Networks Page 14 / 47
Figure 6 : Combining RAPID-PVST and 802.1w after failure of Po1
Figure 6 shows this scenario. When server 1 now wants to send data to server 2, switch B will send it to switch A via Po3 (as
indicated by the MAC address table), which has no connection to the SB9 and will drop the packet. This will not change until
either the MAC address table entry times out (after ~300 seconds) or the server SB9 sends a packet that has been seen by
switch B, whichever happens first.
This scenario shows that RSTP and RAPID-PVST are not compatible in this respect. A worst-case failover time of 300 sec will
not be acceptable.
Running RAPID-PVST on VLAN Trunks while disabling STP at the SB9
When RAPID-PVST is running at the Cisco switches and STP is disabled at the SB9 we have almost the same scenario as
above, where the Cisco switches were running STP and STP was disabled at the SB9. Figure 7 shows this scenario.
Root port
forwarding
Po2
Cisco B
priority 4096 for all vlans
Alternate
discarding
On all trunks:
VLAN 1 native
VLAN 10 tagged
VLAN 20 tagged
Cisco A
priority 0 for all vlans
SB9
STP disabled
Designated port
forwarding
Po1
Po1
Designated port
forwarding
Po3Po3
Po2
Figure 7 : RAPID-PVST while STP is disabled at SB9
When the Po1 link fails, the Po2 of switch B will stop receiving BPDUs. After three times the “hello” interval, the switch will
change the state of port Po2 to “learning” and will then follow the normal state machine so that the convergence time is the
same as with 802.1D.
Since the RSTP cannot operate with the proposal/agreement mechanism on this link, root changes will also be relatively slow
within all the VLANs that are running on the trunks to the SB9.
White Paper Issue: October 2006 Integration of BX600 SB9 Switches in Cisco Networks Page 15 / 47
2.4.2 Recommended Solution
As discussed earlier, there are a number of different combinations of STP protocols that can be selected when integrating SB9
switches into Cisco networks. Although using MSTP between the Cisco and the SB9 would be the best solution, it will not be
discussed further in this paper because MSTP is so very unusual in Cisco networks. If you were to run MSTP (802.1s) on the
SB9 switches while using STP or RSTP at the Cisco switches, MSTP would fall back to RSTP and STP respectively.
The resulting and possible solutions are shown in Table 4.
SB9 Switch
802.1D 802.1w No STP
Cisco
Switch
Table 4 : Possible STP combinations when using VLAN Trunks
* SB9 firmware >1.14 required
The recommended solution when running STP over VLAN trunks between Cisco and SB9 switches is to disable STP completely
at the SB9 and run the STP or RSTP protocol at the Cisco switches (see Figure 4 and Figure 7).
When the SB9 is connected to Cisco switches without VLAN trunks, the preferred solution is RSTP, because this would lead to
the shortest failover times.
Caution: In order to avoid loops in the network, please be sure that the VLAN configuration on both uplinks is
Caution: There is a significant difference between disabling STP on the SB9 globally and for each interface:
If STP is disabled for one interface BPDUs are neither sent nor bridged. This behavior may lead to
Caution: When running STP on an SB9 it is important to enable STP at all ports, especially when creating port-
2.4.3 Configuration with VLAN Trunks
You set up the scenario shown in Figure 8 by performing the following steps:
• Step 1: Configure the switches
• Step 2: Verify the configuration
Cisco A
priority 0 for all vlans
PVST+ Ok*Ok Ok
RAPIDPVST
the same. Misconfiguration may lead to unidirectional links and to network loops!
network loops.
When STP is disabled globally BPDUs are bridged. This is needed in the recommended scenarios.
channels: this is not the default and must be enabled manually.
Designated port
forwarding
with restrictions
(Problems with TCN)
Designated port
Po3Po3
Po1
Gi 0/1Gi 0/2
Gi 0/1
forwarding
Gi 0/23
Gi 0/24
with restrictions
(Problems with TCN)
Root port
forwarding
Gi 0/23
Gi 0/24
Po2
Alternate
discarding
Ok
Cisco B
priority 4096 for all vlans
Gi 0/1
SB9
STP disabled
0/12
0/11
Po1
Po2
0/13
0/14
On all trunks:
VLAN 1 native
VLAN 10 tagged
VLAN 20 tagged
Figure 8 : Configuration example RAPID-PVST while STP is disabled at SB9
Step 1: Configure the switches
! SB9 configuration
!
! Disable STP for the whole switch
! (This command is normally not displayed)
no spanning-tree
Loading...
+ 32 hidden pages
You need points to download manuals.
1 point = 1 manual.
You can buy points or you can get point for every manual you upload.