Cisco Nexus B22HP Design And Deployment Manual
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 1 of 41
Cisco Nexus B22HP
Design and Deployment Guide
September, 2011
For further information, questions and comments please contact ccbu-pricing@cisco.com
Design and Deployment Guide
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 2 of 41
Contents
Introduction .............................................................................................................................................................. 3
Network Diagram ..................................................................................................................................................... 3
Hardware Installation............................................................................................................................................... 5
Fabric Extender Management Model ...................................................................................................................... 5
Fabric Connectivity Options ................................................................................................................................... 6
Static Pinning Fabric Interface Connection ........................................................................................................... 6
PortChannel Fabric Interface Connection ............................................................................................................. 7
Configuring a Fabric PortChannel ......................................................................................................................... 8
Virtual PortChannel Connection ........................................................................................................................... 10
Configuring a vPC ............................................................................................................................................... 10
Server Network Teaming ................................................................................................................................ ....... 13
Creating Host-Side vPC for Server Links with LACP .......................................................................................... 13
Configuring the HP Blade Server (Microsoft Windows 2008 Release 2) ............................................................. 14
Fibre Channel over Ethernet ................................................................................................................................. 17
Configuring FCoE ............................................................................................................................................... 17
Configuring the Cisco Nexus 5000 Series and B22HP for FCoE ........................................................................ 19
Debug Commands ................................................................................................................................................. 24
show fex .............................................................................................................................................................. 24
show fex detail .................................................................................................................................................... 25
show interface brief ............................................................................................................................................. 26
show interface ethernet 103/1/1 .......................................................................................................................... 28
show vlan ............................................................................................................................................................ 29
show interface fex-fabric ..................................................................................................................................... 30
Cisco Nexus Configurations ................................................................................................................................. 31
Cisco Nexus 5000 Series Switch 1 Configuration ............................................................................................... 31
Cisco Nexus 5000 Series Switch 2 Configuration ............................................................................................... 35
Conclusion ............................................................................................................................................................. 40
For More Information ............................................................................................................................................. 41
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 3 of 41
Introduction
The Cisco Nexus® B22 Blade Fabric Extender for HP (Cisco Nexus B22HP) provides an extension of the Cisco
Nexus switch fabric to the HP server edge. Logically, it behaves like a remote line card to a parent Cisco Nexus
5000 Series Switch. The fabric extender and the parent Cisco Nexus 5000 Series Switch together form a
distributed modular system. The Cisco Nexus B22HP forwards all traffic to the parent Cisco Nexus 5000 Series
Switch over eight 10 Gigabit Ethernet uplinks. Low-cost uplink connections of up to 10 meters can be made with
copper Twinax cable, and longer connections of up to 100 meters can use the Cisco Fabric Extender Transceiver
(FET-10G). Standard 10-Gbps optics such as short reach (SR) and long reach (LR) are also supported. Downlinks
to each server are auto negotiating for 1 and 10 Gigabit Ethernet and work with all HP Ethernet and converged
network adapter (CNA) mezzanines, allowing customers a choice of Ethernet, Fibre Channel over Ethernet
(FCoE), or Small Computer System Interface over IP (iSCSI) connections. Because the Cisco Nexus B22 is a
transparent extension of a Cisco Nexus 5000 Series Switch, traffic can be switched according to policies
established by the Cisco Nexus 5000 Series Switch with a single point of management.
The Cisco Nexus B22 provides the following benefits:
●
Highly scalable, consistent server access: This distributed modular system creates a scalable server access
environment with no reliance on Spanning Tree Protocol and with consistency between blade and rack
servers.
●
Simplified operations: The availability of one single point of management and policy enforcement using
upstream Cisco Nexus 5000 Series Switches eases the commissioning and decommissioning of blades
through zero-touch installation and automatic configuration of fabric extenders.
●
Increased business benefits: Consolidation, reduced cabling, investment protection through feature
inheritance from the parent switch, and the capability to add functions without the need for a major
equipment upgrade of server-attached infrastructure all contribute to reduced operating expenses (OpEx)
and capital expenditures (CapEx).
Each member of the Cisco Nexus B22 integrates into the I/O module slot of a third-party blade chassis, drawing
both power and cooling from the blade chassis itself.
Network Diagram
Figure 1 presents a sample network topology that can be built using the Cisco Nexus B22HP, Cisco Nexus 2000
Series Fabric Extenders, and Cisco Nexus 5000 Series Switches. In this topology, the Cisco Nexus 5000 Series
serves as the parent switch, performing all packet switching and policy enforcement for the entire distributed
modular system. The Cisco Nexus 5000 Series also serves as the only point of management for both configuration
and monitoring within the domain, making it simple to manage blade server and rack server connections together.
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 4 of 41
Figure 1. Cisco Nexus Virtual Chassis Topology
The Cisco Nexus 5000 Series Switches, along with the Cisco Nexus 2000 Series and Cisco Nexus B22, create a
distributed modular system that unifies the data center architecture. Within this distributed modular system, both
blade servers and rack servers are managed identically. This approach allows the use of the same business and
technical processes and procedures for the network when addressing the computing environment.
The left most blade chassis in Figure 1 contains dual Cisco Nexus B22HP fabric extenders. Each Cisco Nexus
B22HP is singly attached to a parent Cisco Nexus 5500 switch platform, a connection mode referred to as straight
through mode. The fabric links can either be statically pinned or put into a PortChannel. This connection mode
helps ensure that all data packets from a particular Cisco Nexus B22 enter the same parent Cisco Nexus 5500
switch platform. This approach may be necessary when certain types of traffic must be restricted to either the left
or right Cisco Nexus 5500 switch platform: for instance, to maintain SAN A and SAN B separation. Also, in this
example the connections to individual blade servers are in active-standby mode, which helps ensure traffic flow
consistency but does not fully utilize the server network interface card (NIC) bandwidth.
The second blade chassis from the left in Figure 1 improves on the first with the creation of an Ethernet virtual
PortChannel (vPC) from the blade servers to the Cisco Nexus 5500. This vPC places the Ethernet portion of the
NICs in an active-active configuration, giving increased bandwidth to each host. The FCoE portion of the CNA is
also configured as active-active but maintains SAN A and SAN B separation because each virtual Fibre Channel
(VFC) interface is bound to a particular link at the server. This configuration also achieves high availability through
redundancy, and it can withstand a failure of a Cisco Nexus 5500 switch platform, a Cisco Nexus B22HP, or any
connecting cable. This topology is widely used in FCoE deployments.
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 5 of 41
Card
IOM
LAN on motherboard (LOM)
IOM bays 1 and 2
Mezzanine card 1
IOM bays 3 and 4
Mezzanine card 2
IOM bay 5 and 6
The third blade chassis from the left in Figure 1 contains Cisco Nexus B22HP fabric extenders that connect to both
Cisco Nexus 5500 switch platforms through vPC for redundancy. In this configuration, active-active load balancing
using vPC from the blade server to the Cisco Nexus 5500 switch platform cannot be enabled. However, the servers
can still be dual-homed with active-standby or active-active transmit-load-balancing (TLB) teaming. This topology is
only for Ethernet traffic because SAN A and SAN B separation between the fabric extender and the parent switch
is necessary.
The last two setups illustrate how rack mount servers can connect to the same Cisco Nexus parent switch using
rack-mount Cisco Nexus 2000 Series Fabric Extenders. The topology for blade servers and rack-mount servers
can be identical if desired.
Hardware Installation
Installation of the Cisco Nexus B22HP in the rear of the HP BladeSystem c7000 chassis is similar to the installation
of other I/O modules (IOMs). The layout of the HP BladeSystem c7000 chassis, server types, and mezzanine cards
used determine the slots that should be populated with the Cisco Nexus B22HP for 1 and 10 Gigabit Ethernet
connectivity. Table 1 summarizes the typical options for half-height servers using dual-port 10 Gigabit Ethernet
devices.
Table 1. Mapping of HP BladeSystem c7000 Half-Height Server Mezzanine Card to IOM Bay
After the Cisco Nexus B22HP fabric extenders are installed, the onboard administrator (OA) should be updated to
at least Version 3.5 to help ensure that all functions and graphics are present. No configuration is required from the
chassis onboard administrator.
Fabric Extender Management Model
The Cisco Nexus fabric extenders are managed by a parent switch through the fabric interfaces using a zero-touch
configuration model. The switch discovers the fabric extender by a using detection protocol.
After discovery, if the fabric extender has been correctly associated with the parent switch, the following operations
are performed:
1. The switch checks the software image compatibility and upgrades the fabric extender if necessary.
2. The switch and fabric extender establish in-band IP connectivity with each other. The switch assigns an IP
address in the range of loopback addresses (127.15.1.0/24) to the fabric extender to avoid conflicts with IP
addresses that might be in use on the network.
3. The switch pushes the configuration data to the fabric extender. The fabric extender does not store any
configuration locally.
4. The fabric extender updates the switch with its operational status. All fabric extender information is displayed
using the switch commands for monitoring and troubleshooting.
This management model allows fabric extender modules to be added without adding management points or
complexity. Software image and configuration management is also automatically handled without user intervention.
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 6 of 41
Interface
Fabric Link
1, 2, 3, 4, 5, 6, 7, and 8
Fabric link 1
9, 10, 11, 12, 13, 14, 15, and 16
Fabric link 2
Interface
Fabric Link
1, 2, 3, and 4
Fabric link 1
5, 6, 7, and 8
Fabric link 2
9, 10, 11, and 12
Fabric link 3
13,14,15, and 16
Fabric link 4
Fabric Connectivity Options
The Cisco Nexus B22HP creates a distributed modular chassis with the Cisco Nexus parent switch after a fabric
connection has been made over standard 10-Gbps cabling. This connection can be accomplished using any of the
following types of interconnects:
●
Cisco passive direct-attach cables (1M, 3M, or 5M)
●
Cisco active direct-attach cables (7M or 10M)
●
Cisco standard Enhanced Small Form-Factor Pluggable (SFP+) optics (SR or LR)
●
Cisco Fabric Extender Transceivers
After the fabric links have been physically established, the logical configuration of the links needs to be performed.
There are two methods of connection for the fabric links to the Cisco Nexus B22HP:
●
Static pinning fabric interface connection
●
PortChannel fabric interface connection
Static Pinning Fabric Interface Connection
Static Pinning is the default method of connection between the fabric extender and the Cisco Nexus parent switch.
In this mode of operation, a deterministic relationship exists between the host interfaces and the upstream parent
with up to eight fabric interfaces. These fabric interfaces are equally divided among the 16 server-side host ports. If
fewer fabric ports are allocated, then more server ports are assigned to a single fabric link. The advantage of this
configuration is that the traffic path and the amount of allocated bandwidth are always known for a particular set of
servers.
Since static pinning will group host-side ports into individual fabric links, you should understand its relationship and
how ports are grouped. The size of the port groups is determined by the number of host ports divided by the
max-link parameter value. Thus, if the max-link parameter is set to 2, then eight host ports would be assigned to
each link. The interfaces will be grouped in ascending order starting from the interface 1. Thus, interfaces 1 to 8 will
be pinned to one fabric link, and interfaces 9 to 16 will be pinned to a different interface (Table 2).
Table 2. Interface Assignment with Two Fabric Links
Table 3 summarizes the assignment with four fabric links with the max-link parameter set to 4, the interfaces are
divided into four groups.
Table 3. Interface Assignment with Two Fabric Links
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 7 of 41
Interface
Fabric Link
1 and 2
Fabric link 1
3 and 4
Fabric link 2
5 and 6
Fabric link 3
7 and 8
Fabric link 4
9 and 10
Fabric link 5
11 and 12
Fabric link 6
13 and 14
Fabric link 7
15 and 16
Fabric link 8
Table 4 summarizes the assignment of eight fabric links with the max-link parameter set to 8; the interfaces are
divided into eight groups.
Table 4. Interface Assignment with Two Fabric Links
Note: The assignment of the host-side ports is always based on the configured max-link parameter and not the
actual physical number of fabric ports connected. Be sure to match the max-link parameter with the actual number
of physical links used.
Note: The relationship of host-side ports to parent switch fabric ports is static. If a fabric interface fails, all its
associated host interfaces are brought down and will remain down until the fabric interface is restored.
PortChannel Fabric Interface Connection
The PortChannel fabric interface provides an alternative method of connection between the parent switch and the
Cisco Nexus B22HP fabric extender. In this mode of operation, the physical fabric links are bundled into a single
logical channel. This approach prevents a single fabric interconnect link loss from disrupting traffic to any one
server. The total bandwidth of the logical channel is shared by all the servers, and traffic is spread across the
members through the use of a hash algorithm.
●
For a Layer 2 frame, the switch uses the source and destination MAC addresses.
●
For a Layer 3 frame, the switch uses the source and destination MAC addresses and the source and
destination IP addresses.
Since both redundancy and increased bandwidth are possible, configuration of the fabric links on a PortChannel is
the most popular connection option.
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 8 of 41
Figure 2 shows PortChannel designs.
Note: A fabric interface that fails in the PortChannel does not trigger a change to the host interfaces. Traffic is
automatically redistributed across the remaining links in the PortChannel fabric interface.
Figure 2. PortChannel Designs
Configuring a Fabric PortChannel
1. Log into the first parent switch and enter into configuration mode.
Nexus 5000 Switch
login: admin
Password:
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2011, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
N5548-Bottom# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
N5548-Bottom(config)#
2. Enable the fabric extender feature.
N5548-Bottom(config)#
N5548-Bottom(config)# feature fex
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 9 of 41
N5548-Bottom(config)#
3. Logically create the fabric extender.
N5548-Bottom(config)#
N5548-Bottom(config)# fex 103
N5548-Bottom(config-fex)#
4. Create the PortChannel, change the port mode, and associate the fabric extender with the PortChannel.
N5548-Bottom(config-if)# interface port-channel 3
N5548-Bottom(config-if)# switchport mode fex-fabric
N5548-Bottom(config-if)# fex associate 103
N5548-Bottom(config-if)#
5. Assign the Cisco Nexus parent switch ports to the PortChannel.
N5548-Bottom(config-if)# interface ethernet 1/1
N5548-Bottom(config-if)# switchport mode fex-fabric
N5548-Bottom(config-if)# fex associate 103
N5548-Bottom(config-if)# channel-group 3
N5548-Bottom(config-if)# interface ethernet 1/2
N5548-Bottom(config-if)# switchport mode fex-fabric
N5548-Bottom(config-if)# fex associate 103
N5548-Bottom(config-if)# channel-group 3
6. Repeat the steps on the second Cisco Nexus 5000 Series Switch connected to the fabric extender in
interconnect bay 4.
N5548-Top# configure terminal
N5548-Top(config)# feature fex
N5548-Top(config)# fex 104
N5548-Top(config-if)# interface port-channel 4
N5548-Top(config-if)# switchport mode fex-fabric
N5548-Top(config-if)# fex associate 104
N5548-Top(config-if)# interface ethernet 1/1
N5548-Top(config-if)# switchport mode fex-fabric
N5548-Top(config-if)# fex associate 104
N5548-Top(config-if)# channel-group 4
N5548-Top(config-if)# interface ethernet 1/2
N5548-Top(config-if)# switchport mode fex-fabric
N5548-Top(config-if)# fex associate 104
N5548-Top(config-if)# channel-group 4
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 10 of 41
7. Verify that the Cisco Nexus B22HP is up and running.
switch(config-if)# show fex
FEX FEX FEX FEX
Number Description State Model Serial
------------------------------------------------------------------------
103 FEX0103 Online N2K-B22HP-P FOC1515ZZU4
Virtual PortChannel Connection
vPCs allow links that are physically connected to two different Cisco Nexus switches to form a PortChannel to a
downstream device. The downstream device can be a switch, a server, or any other networking device that
supports IEEE 802.3ad PortChannels. vPC technology enables networks to be designed with multiple links for
redundancy while also allowing those links to connect to different endpoints for added resiliency (Figure 3).
More information about vPC technology can be found at
http://www.cisco.com/en/US/products/ps9670/products_implementation_design_guides_list.html.
Figure 3. Blade Server Configuration Options
Configuring a vPC
Enable vPC (this configuration should be implemented on both switches in parallel):
1. Enable the vPC feature
2. Create the vPC domain
3. Configure the peer keepalive link
4. Configure the vPC peer link PortChannel
5. Check the status of vPC
1. Enable the vPC feature.
N5548-Bottom# configure terminal
N5548-Bottom(config)# feature vpc
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 11 of 41
N5548-Top# configure terminal
N5548-Top(config)# feature vpc
2. Create the vPC domain (should be unique within network).
N5548-Bottom(config)# vpc domain 5
N5548-Top(config)# vpc domain 5
3. Configure the peer keepalive link over the management network.
N5548-Bottom(config-vpc-domain)# peer-keepalive destination 172.25.182.104 source
172.25.182.103
Note:
--------:: Management VRF will be used as the default VRF ::--------
N5548-Top(config-vpc-domain)# peer-keepalive destination 172.25.182.103 source
172.25.182.104
Note:
--------:: Management VRF will be used as the default VRF ::--------
4. Configure the vPC peer link.
N5548-Bottom# interface port-channel 20
N5548-Bottom(config-if)# interface ethernet 1/9
N5548-Bottom(config-if)# channel-group 20
N5548-Bottom(config-if)# interface ethernet 1/10
N5548-Bottom(config-if)# channel-group 20
N5548-Bottom(config-if)# interface port-channel 20
N5548-Bottom(config-if)# vpc peer-link
Please note that spanning tree port type is changed to “network” port type on vPC
peer-link.
This will enable spanning tree Bridge Assurance on vPC peer-link provided the STP
Bridge Assurance
(which is enabled by default) is not disabled.
N5548-Bottom(config-if)#
N5548-Top# interface port-channel 20
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 12 of 41
N5548-Top (config-if)# interface ethernet 1/9
N5548-Top(config-if)# channel-group 20
N5548-Top(config-if)# interface ethernet 1/10
N5548-Top(config-if)# channel-group 20
N5548-Top(config-if)# interface port-channel 20
N5548-Top(config-if)# vpc peer-link
Please note that spanning tree port type is changed to “network” port type on vPC
peer-link.
This will enable spanning tree Bridge Assurance on vPC peer-link provided the STP
Bridge Assurance
(which is enabled by default) is not disabled.
N5548-Bottom(config-if)#
5. Check the vPC status.
N5548-Bottom(config-if)# show vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 5
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status: success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 0
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------
1 Po20 up 1
N5548-Bottom(config-if)#
Now the two switches have been configured to support vPC links to other devices. These connections can be used
for upstream links to the data center core. These vPC links can be used for connections to hosts in the data center,
allowing additional bandwidth and redundant links.
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 13 of 41
Server Network Teaming
Server NIC teaming provides an additional layer of redundancy to servers. It allows multiple links to be available,
for redundancy. In the blade server environment, server network teaming was typically limited to active-standby
configurations and could not provide active-active links, because active-active links required EtherChannel or Link
Aggregation Control Protocol (LACP) connection to a single switch. Since the Cisco Nexus B22HP fabric extender
is an extension of the parent switch, EtherChannel or LACP connections can be created between the blade server
and the virtual chassis. Dual Cisco Nexus 5000 Series switches can be used with vPC for additional switch
redundancy while providing active-active links to servers, thus enabling aggregate 40-Gbps bandwidth with dual
links (Figure 4).
Figure 4. Fabric Link and Server Topologies
Creating Host-Side vPC for Server Links with LACP
1. Enable LACP on both parent switches.
5548-Bottom (config)# feature lacp
2. Create the blade server vPC and add the member interface.
nexus-5548-Bottom# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
5548-Bottom(config)# interface port-channel 201
5548-Bottom(config-if)# vpc 201
5548-Bottom(config-if)# switchport mode access
5548-Bottom(config-if)# no shutdown
5548-Bottom(config-if)# interface ethernet 103/1/1
5548-Bottom(config-if)# channel-group 201 mode active
nexus-5558-Top# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
5548-Top(config)# interface port-channel 201
5548-Top(config-if)# vpc 201
5548-Top(config-if)# switchport mode access
5548-Top(config-if)# no shutdown
5548-Top(config-if)# interface ethernet 104/1/1
5548-Top(config-if)# channel-group 201 mode active
Loading...