Dell 40G, MXL 10 User Manual

Victor Teeter

Network Enabled Solutions Team

Stacking Dell Force10 MXL 10/40G

Blade Switches

Stacking Dell Force10 MXL 10/40G Blade Switches v1.1

This document is for informational purposes only and may contain typographical errors and technical inaccuracies. The content is provided as is, without express or implied warranties of any kind.

© 2012 Dell Inc. All rights reserved. Dell and its affiliates cannot be responsible for errors or omissions in typography or photography. Dell, the Dell logo, and PowerEdge are trademarks of Dell Inc. Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and names or their products. Dell disclaims proprietary interest in the marks and names of others.

November 2012| Rev 1.1

Stacking Dell Force10 MXL 10/40G Switches

Contents

Introduction ............................................................................................................. 3

Redundancy ........................................................................................................... 4

Failover roles ......................................................................................................... 4

Stacking LAG .......................................................................................................... 4

Enablement of LAG/LACP on blade servers ..................................................................... 4

Meta-data ............................................................................................................. 4

Restoring factory defaults .......................................................................................... 5

Constant stack configuration ...................................................................................... 5

Bare Metal Provisioning ............................................................................................. 6

Identifying physical units and ports in a stack .................................................................... 6

Port numbering ....................................................................................................... 7

Combining split 10GbE SFP+ ports into a single 40GbE QSFP+ port ......................................... 8

Master selection criteria .............................................................................................. 9

Manually selecting a Master during initial stack setup ........................................................ 9

Stacking scenarios .................................................................................................... 10

Creating the MXL stack ........................................................................................... 13

Adding new members to an existing stack .................................................................... 17

Updating firmware on a stack ................................................................................... 17

Un-stacking (removing member units) ......................................................................... 20

Renumbering stack units ......................................................................................... 21

Appendix A – Stacking scenarios ...................................................................................... 23

Appendix B – Port mapping ............................................................................................ 24

Appendix C - Network switch versions ............................................................................... 25

Figures

Figure 1. Dell Force10 MXL (10/40G Ethernet) ................................................................... 3

Figure 2. I/O module overview from the CMC .................................................................... 6

Figure 3. Port mapping for QSFP+ ports............................................................................ 7

Figure 4. QSFP+ two 40GbE port expansion for the MXL ...................................................... 10

Figure 5. Stacking multiple MXL switches in a single M1000e ................................................ 12

Figure 6. Stacking MXL switches across multiple M1000e chassis using two ports between members 12

Stacking Dell Force10 MXL 10/40G Switches

Introduction

Numerous Dell™ switches include a stacking feature that allows multiple physical switches to operate as a single logical switch, providing a consolidated interface for management. The Dell Force10 MXL 10/40Gb switch is the latest Dell switch with this feature. Up to six Dell Force10 MXL switches can be connected in a single stack using QSFP+ (40Gb) ports.

A single switch (Master) in the stack controls all switches in the stack thereby allowing the user to manage and configure all member switches and ports using one IP address. This IP address is copied from the Master to the Standby when the Standby is created. If for any reason the Master fails and the Standby takes over as the Master, the IP address of the stack will remain the same, allowing continuous management of the stack.

The new Master unit will also continue to use the original Master unit’s MAC address which helps to reduce disruptions to the network. When a failed Master re-joins the stack, it does so as a member (not a Master) unless a new Master has not had time to be elected.

Up to four QSFP+ (40Gb) ports can be used for stacking on any member switch. Only two scenarios support using four ports. One uses all four ports in a daisy-chain stack topology containing only two switch members. The other uses a ring topology with two links between each member (see Figure 6). Three or four QSFP+ ports may also be used between each member in a daisy-chain topology containing only two member switches. The typical scenario uses one or two QSFP+ ports to connect stack members in a ring topology while allowing available QSFP+ ports for uplinks. Dell recommends a ring topology (connecting the first and last members to create a loop) for a more resilient stack that helps avoid stack splits. Stack splits are covered below in more detail.

This document provides an easy to use step-by-step guide on how to configure stacking for the Dell Force10 MXL (Figure 1).

Dell Force10 MXL (10/40G Ethernet) Figure 1.

Note: The Dell Force10 MXL can only be stacked with other Dell Force10 MXL switches.

Note: Only QSFP+ ports can be used for stacking and they must be in 40Gb mode. Stacking cannot be

enabled on these ports when split into four 10Gb ports.

An important advantage of stacking is that it provides a consolidated interface for management of multiple switches linked together. After a stack is deployed in the network, operators can easily add units to the stack as their port requirements increase, with minimal administrative overhead.

Stacking Dell Force10 MXL 10/40G Switches

Additional stack members can immediately utilize existing configuration information such as routing and switching configurations, VLANs, ACLs, port profiles, and security certificates.

Redundancy

By connecting a cable from the last switch in a stack back to the first switch, the operator ensures that a stack has the protection of redundant paths for control and data traffic. Support for link aggregation groups (LAG) configured across multiple switches provides yet another layer of redundancy as well as adding bandwidth. This means that any single point of failure (a switch or a stack cable failure) will not affect the overall operation of the remaining stack elements. This type of stacking topology is referred

to as a “ring topology”. The other stacking topology where the last and first switch are not connected

is referred to as a daisy-chain topology stack. This type of stack is much less resilient and not recommended in most cases. Both topologies are mentioned several times in this document

Failover roles

If the stack Master fails (for example, loses power), it is removed from the stack topology. The Standby unit detects the loss of peering communication and takes ownership of the stack management, switching from Standby to Master. If a Standby was not set up by the administrator, the stack automatically triggers an election within the remaining units to select a new Master. While a new Master is being selected (either through election or pre-selection), a new Standy is also chosen from the remaining members based on the same criteria as a Master selection (priority, then highest MAC address).

After the former Master switch recovers, despite having a higher priority or MAC address, it does not recover its Master role but instead take the next available role as Standy or Member.

Stacking LAG

When multiple links between stack members are used, the Dell Force10 MXL automatically bundles them into a single logical link, or LAG, providing higher stacking bandwidth and redundancy. The stacking LAG is established automatically without user configuration once all ports used in the LAG are set in stacking mode. The LAG can lose link or gain links simply by removing or inserting the cables.

Enablement of LAG/LACP on blade servers

Another benefit of stacking MXL switches is that it allows a blade server to create LAGs to the stacked switch. For example, if MXL switches are stacked in slots B1 and B2, then NICs on B1 and B2 on a given

server could be configured for LAGs/LACP.

Meta-data

The actual stack configuration used to stack switches during power-up is read from meta-data, not from the startup configuration. It is applied at boot time prior to starting the switch firmware and reading the startup configuration. Stack information shown in the startup and running configurations is simply repeating information from the meta-data for the user’s knowledge. Therefore, if the startup configuration is deleted and the switch is reloaded, it will not clear the stacking configuration and the

stack will remain intact after the reload. In order to remove the stacking configuration, the “no”

command will need to be implemented for each of the stack-unit commands that were used to create the stack.

Stacking Dell Force10 MXL 10/40G Switches

Another way to completely wipe out all configurations and return to factory defaults is to erase the meta-data (NVRAM) and delete the startup configuration by following the instructions in Restoring factory defaults below.

Restoring factory defaults

Usually a standalone MXL can be set to factory defaults simply by deleting the startup configuration and reloading the switch (Step 2). However, once an MXL has been stacked, additional steps are required in order to remove the stacking configuration from the NVRAM meta-data. To completely remove all configurations (including stacking) and return to factory defaults on any stack member:

1. Unplug the stacking cables connecting the switch to the rest of the stack.

2. From the console prompt of the removed switch, type delete startup-configuration and press

enter, then reload the system. If prompted to save the configuration, enter “no”.

3. During reboot, press any key at the second prompt to get to the BOOT_USER # menu (the

bootup process allows 5 seconds to press any key to get to this menu).

4. From the BOOT_USER # prompt, type enable admin to get into administrative mode. The

default password is “ncorerulz”, which brings up the BOOT_ADMIN # prompt.

5. Type nvram erase <enter>. Then type reload <enter>. When the reload completes, the switch

will be at factory defaults.

Constant stack configuration

A minimum stack configuration is always present on a stack-capable switch. Even if a switch has never been stacked with other switches, it is still considered a “stack of 1”, and therefore will always contain “stack-unit” lines in the running configuration. Here is an example configuration of a device that is not stacked.

Note: FTOS starts numbering stack units at zero, so a single stack-unit 0 as shown below is in fact a stack unit of 1.

FTOS#show running-config Current Configuration ... ! Version 8.3.16.0E0 ! Startup-config last updated at Thu May 3 11:49:39 2012 ! boot system stack-unit 0 primary system: A: boot system stack-unit 0 secondary system: A: boot system gateway 172.31.0.1 ! redundancy auto-synchronize full ! hostname FTOS ! username root password 7 d7acc8a1dcd4f698 privilege 15 !

 stack-unit 0 provision MXL-10/40GbE

! interface TenGigabitEthernet 0/1

Stacking Dell Force10 MXL 10/40G Switches

no ip address shutdown ! interface TenGigabitEthernet 0/2 no ip address

--More--

Additional stack-unit lines like the one below may be found in an unstacked MXL if one or more expansion modules are present. In order to use these ports for stacking the quad mode will need to be removed.

stack-unit 0 port (41, 43, 49, 53) portmode quad

Bare Metal Provisioning

The system is configured for Bare Metal Provisioning (BMP) mode when it leaves the factory. Bare Metal Provisioning can be used to help configure a stack or a stand-alone system. With minimum effort, BMP can obtain IP address, running configuration and boot image information from a DHCP server. For more information regarding how to setup BMP, consult the Dell Force10 MXL User Guide.

Identifying physical units and ports in a stack

The Master will always have a blue LED illuminating steadily (not flashing). A stand-alone will also have the blue light illuminated since it is it’s own Master. Standby and Member units in the stack do not illuminate this light.

Note: The CMC shows each Master in the chassis and the fabric (A1, A2, B1, B2, C1, or C2) where it is located. See 0.

I/O module overview from the CMC Figure 2.

Stacking Dell Force10 MXL 10/40G Switches

Port numbering

When installed in a PowerEdge™ M1000e Enclosure, the MXL 10/40GbE external switch ports are numbered 33 to 56 from the bottom to the top of the switch (ports 1-32 are internal 10G ports):

• 40GbE base-module (built-in) ports:

 In 2x40GbE mode of operation (default), the ports are numbered 33 and 37.

• In 8x10GbE mode of operation, the ports are numbered 33 through 40. The 40GbE mode of operation is required for stacking. For information about how to change

from 8x10GbE mode to 40GbE mode, refer to Combining split 10GbE SFP+ ports into a single

40GbE QSFP+ port below.

• 2-Port 40-GbE QSFP+ expansion module operating in the default 8x10GbE mode:

• In expansion slot 0, the ports are numbered 41 to 44 and 45 to 48.

• In expansion slot 1, the ports are numbered 49 to 52 and 53 to 56.

The 40GbE mode of operation is required for stacking. For information about how to change from 8x10GbE mode to 40GbE mode, refer to Combining split 10GbE SFP+ ports into a single

40GbE QSFP+ port below.

• 2-Port 40-GbE QSFP+ expansion module operating in 40GbE mode:

• In expansion slot 0, the ports are numbered 41 and 45.

• In expansion slot 1, the ports are numbered 49 and 53.

• 4-Port 10-GbE SFP+ or 10GBASE-T module (NOT USED FOR STACKING):

• In expansion slot 0, the ports are numbered 41 to 44.

• In expansion slot 1, the ports are numbered 49 to 52.

 Only one 10GBaseT expansion module can be installed per MXL.

Port mapping for QSFP+ ports Figure 3.

See Appendix B – Port mapping for a complete list of port numbers.

+ 18 hidden pages