Dell Active Fabric Manager Deployment Guide
Active Fabric Manager (AFM)
Deployment Guide 2.0
© 2013 Dell Inc. All Rights Reserved.
Trademarks used in this text:
PowerConnect
of Dell Inc.
®
AMD
Devices, Inc.
either trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries.
Red Hat
™
OpenManage
,
®
Intel
Pentium
is a registered trademark and
,
Microsoft
®
Enterprise Linux
®
are registered trademarks of Novell Inc. in the United States and other countries.
Corporation and/or its affiliates.
™
Dell
, the Dell logo,
™
EqualLogic
,
®
®
Xeon
Core
,
,
AMD Opteron
®
Windows
,
®
are registered trademarks of Red Hat, Inc. in the United States and/or other countries.
Windows Server
,
®
,
Citrix
Xen
®
and
®
Dell Boomi
™
Compellent
,
Celeron
™
,
XenServer
Systems, Inc. in the United States and/or other countries.
trademarks or trademarks of VMware, Inc. in the United States or other countries.
™
Dell Precision
,
™
KACE
,
®
are registered trademarks of Intel Corporation in the U.S. and other countries.
AMD Phenom
,
®
Internet Explorer
,
®
and
VMware
™
XenMotion
®
,
™
™
and
vMotion
,
FlexAddress
,
AMD Sempron
®
MS-DOS
,
®
are either registered trademarks or trademarks of Citrix
®
,
vCenter
™
OptiPlex
™
Force10
,
™
®
Windows Vista
,
®
is a registered trademark of Oracle
Oracle
®
,
vCenter SRM
®
is a registered trademark of International
IBM
™
Latitude
,
are trademarks of Advanced Micro
™
Venue
,
PowerEdge
,
™
and
®
Active Directory
and
™
and
vSphere
Business Machines Corporation.
2013 - 12
Rev. A0X
™
Vostro
Red Hat
Novell
PowerVault
,
™
are trademarks
®
are
®
and
®
and
SUSE
®
are registered
™
,
®
Contents
1 Introduction..................................................................................................................................9
Problem: Challenges to Build a Fabric in the Data Center........................................................................................9
Solution: Active Fabric Manager..............................................................................................................................9
2 About AFM..................................................................................................................................11
3 Getting Started...........................................................................................................................13
Designing and Deploying a Fabric..........................................................................................................................13
Designing and Deploying a Fabric Flowchart.........................................................................................................15
4 AFM Site Map............................................................................................................................ 17
5 Supported Fabric Types........................................................................................................... 19
Key Considerations for Designing a Layer 3 with Resiliency (Routed VLT) Fabric.................................................20
Gathering Useful Information for a Layer 3 with Resiliency (Routed VLT) Fabric.................................................. 20
Conventional Core Versus Distributed Core........................................................................................................... 21
Conventional Core............................................................................................................................................ 21
Distributed Core............................................................................................................................................... 22
Key Advantages............................................................................................................................................... 22
Distributed Core Terminology ..........................................................................................................................23
Key Considerations for Designing a Distributed Core......................................................................................24
Gathering Useful Information for a Distributed Core....................................................................................... 25
Selecting a Layer 3 Distributed Core Fabric Design........................................................................................ 26
VLT.......................................................................................................................................................................... 30
Multi-domain VLT............................................................................................................................................. 31
VLT Terminology...............................................................................................................................................31
VLT Fabric Terminology....................................................................................................................................31
VLT Components...............................................................................................................................................32
Typical VLT Topology....................................................................................................................................... 32
Key Considerations for Designing a Layer 2 VLT Fabric.................................................................................. 33
Gathering Useful Information for a Layer 2 VLT Fabric....................................................................................34
Selecting a Layer 2 and Layer 3 with Resiliency (Routed VLT) Fabric Design................................................. 34
6 Designing the Fabric.................................................................................................................49
Network Deployment Summary .............................................................................................................................49
Fabric Configuration Phases and States..........................................................................................................49
Switch Configuration Phases and States.........................................................................................................51
Using the Fabric Design Wizard............................................................................................................................. 51
Fabric Design – Step 1: Fabric Name and Type............................................................................................... 52
Fabric Design – Step 2: Bandwidth and Port Count......................................................................................... 53
Deployment Topology Use Cases.....................................................................................................................55
Fabric Design – Step 3: Deployment Topology.................................................................................................65
Fabric Design – Step 3: Fabric Customization..................................................................................................71
Fabric Design – Step 5: Output.........................................................................................................................72
Fabric Design – Step 6: Summary.................................................................................................................... 76
Importing an Existing Fabric Design....................................................................................................................... 76
Editing and Expanding an Existing Fabric Design ..................................................................................................77
Deleting the Fabric..................................................................................................................................................77
Viewing the Wiring Diagram...................................................................................................................................77
7 Configuring and Deploying the Fabric................................................................................... 79
Fabric Deployment Summary..................................................................................................................................79
Switch Configuration Phases and States.........................................................................................................79
Operations Allowed in Each Fabric State...............................................................................................................80
Using the Pre-deployment Wizard..........................................................................................................................82
Layer 2 VLT Fabric Pre-deployment ................................................................................................................ 82
Layer 3 Distributed Core Fabric Pre-deployment ............................................................................................ 82
Layer 3 with Resiliency (Routed VLT)...............................................................................................................82
Pre-Deployment Configuration.........................................................................................................................83
Protocol Configuration — Layer 2 VLT Fabric: Step 1......................................................................................85
Protocol Configuration — Layer 3 Distributed Core Fabric: Step 1..................................................................92
Protocol Configuration — Layer 3 with Resiliency (Routed VLT) : Step 1........................................................94
Pre-deployment – Step 2: Assign Switch Identities.......................................................................................105
Pre-Deployment – Step 3: Management IP ...................................................................................................106
Pre-Deployment – Step 4: SNMP and CLI Credentials...................................................................................106
Pre-Deployment – Step 5: Software Images ................................................................................................. 107
Pre-Deployment – Step 6: DHCP Integration..................................................................................................107
Pre-Deployment – Step 7: Summary.............................................................................................................. 108
Viewing the DHCP Configuration File................................................................................................................... 109
Deploying and Validating the Fabric.....................................................................................................................109
Deploying the Fabric...................................................................................................................................... 109
Advanced Configuration ................................................................................................................................112
Validation .......................................................................................................................................................115
Viewing Deployment and Validation Status................................................................................................... 117
Custom CLI Configuration..................................................................................................................................... 117
Managing Templates......................................................................................................................................117
Associating Templates...................................................................................................................................119
Adding a Switch-Specific Custom Configuration ..........................................................................................120
Viewing Custom Configuration History...........................................................................................................121
8 Viewing the Fabric.................................................................................................................. 123
Dashboard............................................................................................................................................................ 123
Network Topology.................................................................................................................................................125
Network Topology Tabular View....................................................................................................................125
Network Topology Graphical View................................................................................................................ 126
Fabric Summary ...................................................................................................................................................127
Displaying the Fabric in a Tabular View.........................................................................................................127
Displaying the Fabric in a Graphical View..................................................................................................... 128
Switch Summary...................................................................................................................................................129
9 Troubleshooting.......................................................................................................................131
Ping, Traceroute, SSH, and Telnet........................................................................................................................131
Ping.................................................................................................................................................................131
Traceroute......................................................................................................................................................131
SSH ................................................................................................................................................................131
Telnet..............................................................................................................................................................132
Validation Alarms..................................................................................................................................................132
Deployment and Validation Errors........................................................................................................................134
Pre-deployment Errors................................................................................................................................... 134
Deployment Errors..........................................................................................................................................134
Validation Errors.............................................................................................................................................135
Switch Deployment Status Errors.........................................................................................................................138
TFTP/FTP Error......................................................................................................................................................143
Validating Connectivity to the ToR........................................................................................................................143
10 Alerts and Events.................................................................................................................. 145
Current — Active Alerts....................................................................................................................................... 145
Historical — Alerts and Event History..................................................................................................................147
11 Performance Management................................................................................................. 149
Network Performance Management....................................................................................................................149
Fabric Performance Management........................................................................................................................150
Switch Performance Management...................................................................................................................... 150
Port Performance Management...........................................................................................................................151
Detailed Port Performance Management............................................................................................................ 151
Data Collection..................................................................................................................................................... 152
Threshold Settings................................................................................................................................................153
Reports..................................................................................................................................................................154
Creating New Reports....................................................................................................................................154
Editing Reports............................................................................................................................................... 155
Running Reports.............................................................................................................................................155
Duplicating Reports........................................................................................................................................155
Deleting Reports.............................................................................................................................................156
12 Maintenance..........................................................................................................................157
Back Up Switch.................................................................................................................................................... 157
Restoring a Switch Configuration ................................................................................................................. 157
Deleting a Backup Configuration................................................................................................................... 157
Editing Description......................................................................................................................................... 158
Updating the Switch Software..............................................................................................................................158
Replacing a Switch...............................................................................................................................................158
Step 1: Decommission a Switch.....................................................................................................................158
Step 2: Replacing a Switch.............................................................................................................................159
Step 3: Deploy Switch.................................................................................................................................... 160
Updating the AFM ................................................................................................................................................ 160
Updating the AFM Server...............................................................................................................................160
Activating the AFM Standby Partition............................................................................................................161
13 Jobs......................................................................................................................................... 163
Displaying Job Results..........................................................................................................................................163
Scheduling Jobs................................................................................................................................................... 163
Switch Backup .............................................................................................................................................. 164
Switch Software Updates.............................................................................................................................. 164
Switch Software Activation........................................................................................................................... 165
Scheduling Switch Software Updates........................................................................................................... 166
Activating Standby Partition Software ..........................................................................................................167
Scheduling a Back Up Switch Configuration ................................................................................................ 167
14 Administration........................................................................................................................169
Administrative Settings.........................................................................................................................................169
Active Link Settings........................................................................................................................................169
CLI Credentials............................................................................................................................................... 171
Client Settings................................................................................................................................................ 171
Data Retention Settings................................................................................................................................. 172
DHCP Server Settings.................................................................................................................................... 172
NTP Server Settings.......................................................................................................................................172
SMTP Email ................................................................................................................................................... 173
SNMP Configuration...................................................................................................................................... 173
Syslog Server IP Addresses...........................................................................................................................173
System Information........................................................................................................................................ 173
TFTP/FTP Settings.......................................................................................................................................... 174
Managing User Accounts.....................................................................................................................................174
Adding a User.................................................................................................................................................175
Deleting a User...............................................................................................................................................176
Editing a User................................................................................................................................................. 176
Unlocking a User............................................................................................................................................ 177
Changing Your Password...............................................................................................................................177
Managing User Sessions......................................................................................................................................178
Audit Log...............................................................................................................................................................178
8
1
Introduction
Active Fabric Manager (AFM) is a graphical user interface (GUI) based network automation and orchestration tool that
enables you to design, build, deploy, and optimize a Layer 2 Virtual Link Trunking (VLT), Layer 3 distributed core, and
Layer 3 with Resiliency (Routed VLT) fabric for your current and future capacity requirements. This tool helps you
simplify network operations, automate tasks, and improve efficiency in the data center.
You can monitor performance at the network, fabric, switch, and port level. You can also display additional performance
statistics through AFM using a Dell OpenManage Network Manager (OMNM) server. It automates common network
management operations and provides advanced network element discovery, remote configuration management, and
system health monitoring to proactively alert network administrators to potential network problems. OMNM provides
SOAP based web services to allow 3rd parties to integrate with it. AFM supports Dell Networking S4810, S4820T, S55,
S60, S6000, MXL blade, and Z9000 switches.
Problem: Challenges to Build a Fabric in the Data Center
• How do you design the fabric?
• What kind of switch do you buy?
• Who is going to use Visio® to manually document the fabric, that is, manually document which switch ports connect
to another switch
• Who is going to draw the cables?
• How will I ensure that this fabric design is accurate?
• Who is going to update the fabric design as I change it or expand it?
• Who is going to configure every switch in the fabric and what kind of errors can happen because this is manually
performed?
• How do I keep track of software versions on each switch?
• Who is going to validate every switch in the fabric to verify that they have the correct version of software and
configuration and that the switches are physically connected to the right switches.
Solution: Active Fabric Manager
9
10
2
About AFM
Active Fabric Manager (AFM) is a graphical user interface (GUI) based network automation and orchestration tool that
allows you to design, build, deploy, and optimize a Layer 3 distributed core, Layer 3 with Resiliency (Routed VLT), and
Layer 2 VLT fabric for your current and future capacity requirements. This tool helps you simplify network operations,
automate tasks, and improve efficiency in the data center.
NOTE: Before you begin, review the Getting Started page. For information about the AFM workflow, see Flowchart
for Designing and Deploying a Fabric. To learn how to install the AFM, including instructions on completing the
Initial Setup, see the
• Getting Started
• Fabric Designer Wizard
• Pre-deployment Wizard
• Deploying the Fabric
• Alerts
• Administration
• Performance Management
Active Fabric Manager Installation Guide
.
11
12
Getting Started
This section contains the following topics:
• Designing and Deploying the Fabric
• Flowchart for Designing and Deploying a Fabric
Related links:
• Supported Fabrics
• Designing the Fabric
• AFM Site Map
NOTE:
You can view the
from the Help pull-down menu in the upper right of the screen.
Active Fabric Manager Deployment Guide
3
in the AFM by selecting the Deployment Guide option
Designing and Deploying a Fabric
This section provides an overview of the steps required to design and deploy a fabric, including the information you
need before you begin.
NOTE: If you are using the OpenStack Neutron Managed option, refer to the
After you complete the basic installation of the Active Fabric Manager (AFM), you must configure it. This is done using
the Getting Started configuration wizard at the Home > Getting Started screen. After you complete the installation
process, the AFM automatically launches this wizard. The Getting Started configuration wizard provides launch points
for designing, pre-deploying, and deploying the fabric. Review the steps in the wizard and the online help or (
Deployment Guide
an existing design.
) before you begin. With this wizard, you can also edit and expand an existing fabric design and import
AFM Plug-in for Openstack Guid
AFM
e.
13
Figure 1. Getting Started Wizard
To design and deploy a Layer 2 VLT, Layer 3 distributed core fabric, or Layer 3 with Resiliency (Routed VLT)
1. Gather useful information.
Related links.
– Gather Useful Information for Layer 2 VLT Fabric
– Gathering Useful Information for a Layer 3 Distributed Core Fabric.
– Gathering Useful Information for a Layer 3 with Resiliency (Routed VLT) Fabric
2. Design the fabric.
Related links designing a Layer 2 VLT fabric:
– Overview of VLT
– Key Considerations fo Designing a VLT Fabric
– Selecting a Layer 2 VLT and Layer 3 with Resiliency (Routed VLT) Fabric Design
Related links for designing a Layer 3 distributed core fabric:
– Overview of a Distributed Core
– Terminology
– Designing a Distributed Core
– Selecting a Distributed Core Design
Related links for designing a Layer 3 with Resiliency (Routed VLT):
– Key Considerations for Designing Layer 3 with Resiliency (Routed VLT)
14
– Selecting a Layer 2 VLT and Layer 3 with Resiliency (Routed VLT) Fabric Design
3. Build the physical network.
4. Configure the following settings:
– TFTP/FTP
– SNMP
– CLI Credentials
5. Prepare the Fabric for Deployment
6. Deploy and Validate the Fabric
7. Validate the deployed fabric against the fabric design.
8. Monitor the fabric health and performance. See Performance Management.
NOTE: To provision the fabric, enter the Dell Networking operating system (FTOS) CLI user’s Credentials and
enable the configuration credential for all the switches in the fabric. For information about this topic, see CLI
Credentials.
CAUTION: If you are using a switch that has already been deployed, reset its factory settings in the fabric. The
switch must be in Bare Metal Provision (BMP) mode.
Designing and Deploying a Fabric Flowchart
The following flowchart shows how to design and deploy a new fabric.
Figure 2. Capacity Planning
15
Figure 3. Provisioning
16
AFM Site Map
To help you navigate the AFM user interface use the following site map.
4
Home Getting Started
Wizard
Step 1: Design the
Fabric
Step 2: PreDeployment
Configuration
Step 3: Deploy the
Fabric
Network
Level
Fabric
Level
Switch
Level
Summary
Map
Network View
Graphical and
Tabular View
Summary
Fabric View
Summary
Device View
Graphical and
Tabular View
Dashboard
Alerts and
Events
Current
Historical
Alerts and
Events
Current
Historical
Alerts and
Events
Current
Historical
Performance
Average
Bandwidth
Utilization
Link Usage
Switch
Statistics
Performance
Average
Bandwidth
Utilization
Link Usage
Switch
Statistics
Performance
Switch and Port
Real-time and
Historical data
Design Fabric
New Fabric
Edit Fabric
Delete Fabric
View Wiring Plan
Maintenance
Software Updates
Backup and
Restore
Troubleshooting
Ping
SSH
Traceroute
Telnet
Configure and
Deploy Fabric
Deploy Fabric
Pre-deployment
Configuration
Deploy and Validate
View DHCP
Configuration
Errors
CLI Configuration
View DHCP
configuration files
Manage Templates
Associate Templates
Custom
Configuration
View Custom
Configuration History
View Wiring Plan
Replace a Switch
Decommission
Switch
Replace Switch
Deploy Switch
17
Jobs Job Results Schedule Jobs
Backup
Switch
Configuration
Files
Update switch
software
Active
Software
Administration Audit Log Administration
Active Link
Settings
CLI
Credentials
Client Settings
Data Retention
Settings
DHCP Server
Settings
NTP Server
Settings
Email Settings
Syslog IP
Addresses
SNMP
Configuration
System
Information
TFTP/FTP
Settings
Data Collection
Schedule data
collection
Edit threshold
User Accounts
Add User
Delete User
Edit User
Unlocking User
Reports
Create
Edit
Delete
Duplicate
Run
User Sessions
Display active
AFM users
Terminate users’
sessions
AFM Server Upgrade
AFM Server Upgrade
AFM Server Backup
18
5
Supported Fabric Types
The fabric design wizard defines the basic configuration for a Layer 2 VLT, Layer 3 distributed core, and Layer 3 with
Resiliency (Routed VLT) fabric.
• Use the Layer 3 distributed core fabric for large fabric deployments. For information about distributed core fabrics,
see Conventional Core Versus Distributed Core and Selecting a Layer 3 Distributed Core Fabric Design.
• Use the Layer 2 VLT fabric for workload migration over virtualized environments. For information about Layer 2
fabrics, see VLT and Selecting a Layer 2 VLT and Layer 3 with Resiliency (Routed VLT) Fabric Design.
• Use the Layer 3 with Resiliency (Routed VLT) fabric to extend equal cost multi-pathing capabilities. For information
about supported tiers, see Selecting a Layer 2 VLT and Layer 3 with Resiliency (Routed VLT) Fabric Design.
See also Deployment Topology Use Cases. For information about tiers, see Deployment Topology.
To design a fabric based on the capacity requirements for your current and future needs, use the fabric design wizard at
the Network > Configure Fabric > Design New Fabric screen. When you first start AFM, it starts the Getting Started
configuration wizard in the Welcome to Active Fabric Manager screen.
Figure 4. Getting Started: Welcome to Active Fabric Manager Screen
19
Key Considerations for Designing a Layer 3 with Resiliency (Routed VLT) Fabric
Use the Layer 3 with Resiliency (Routed VLT) fabric to extend equal cost multi-pathing capabilities. When designing a
Layer 3 with Resiliency (Routed VLT) fabric, consider the following:
• You can deploy up to 10 fabrics. However, the fabrics do not communicate with each other.
• AFM manages Dell Networking S4810, S4820T, S6000, and Z9000 switches.
CAUTION: If you are already using a deployed switch, you must reset the factory settings. The switch must be in
BMP mode.
For more information on BMP, see DHCP Integration and the
S4820T, S6000, and Z9000 switches at https://www.force10networks.com/CSPortal20/KnowledgeBase/
Documentation.aspx.
The number and type of switches in a Layer 3 with Resiliency (Routed VLT) fabric are based on the following:
• The number of current uplinks (minimum of 2) and downlinks for the access switches.
• The number of planned edge ports (future uplinks and downlinks) for the access switches.
• Whether the access switches need to act as a ToR or access.
• Fabric interlink bandwidth (the links between the aggregation and access switches).
• Downlinks which can be 1Gb, 10Gb, or 40 Gb.
• The fabric interlink bandwidth, 10 Gb or 40 Gb, is fixed and based on the fabric type.
CAUTION: If you do not specify additional links in the fabric design for future expansion in the Bandwidth and Port
Count screen you can only expand the downlinks on the existing fabric.
For information on how to expand a fabric, see Editing and Expanding an Existing Fabric Design. For information about
tiers, see Deployment Topology See also Deployment Topology Use Cases.
FTOS Configuration Guide
for the Dell Networking S4810,
Gathering Useful Information for a Layer 3 with Resiliency (Routed VLT) Fabric
To gather useful information for a Layer 3 with Resiliency (Routed VLT) fabric before you begin:
• Obtain the CSV file that contains the system MAC addresses, service tag and serial numbers for each switch
provided from Dell manufacturing or manually enter this information.
• Obtain the location of the switches, including the rack and row number from your network administrator or network
operator.
• Obtain the remote Trivial File Transfer Protocol (TFTP) / File Transfer Protocol (FTP) address from your network
administrator or network operator. To specify a TFTP/FTP site, go to Administration > Settings >TFTP/FTP screen. For
information about which software packages to use, see the Release Notes.
• Download the software image for each type of switch in the fabric. Each type of switch must use the same version of
the software image within the fabric. Place the software images on the TFTP/FTP site so that the switches can
install the appropriate FTOS software image and configuration file.
• Obtain the Dynamic Host Configuration Protocol (DHCP) server address to use for the fabric from your DHCP
network administrator or network operator. If a remote DHCP server is not available, AFM also provides a local
DHCP. The DHCP server must be in the same subnet where the switches are located. After you power cycle the
20
switches, the switches communicate with the DHCP server to obtain a management IP Address based on the system
MAC Address. The DHCP server contains information about where to load the correct software image configuration
file for each type of switch from the TFTP/FTP site during BMP. For information about BMP, see DHCP Integration.
• Obtain the pool of IP addresses for the management port for each switch in the fabric.
• Obtain IP addresses (must be an even number) for the uplink configuration from the ISP service. The uplink port
number range is based on whether a 10 Gb or 40 Gb bandwidth is selected.
– For 10 Gb uplink bandwidth, AFM supports 2 to 32 uplinks.
– For 40 Gb uplink bandwidth, AFM supports 2 to 8 uplinks.
• Obtain IP addresses or VLAN ID for the downlink configuration for connecting to the server or ToR.
• Gather protocol configuration for uplinks and downlinks.
Conventional Core Versus Distributed Core
This section describes the differences between a conventional core and a distributed core.
Conventional Core
A conventional core is a three-tier network that is typically chassis based and is composed of the following:
• Core — The core layer routes traffic to and from the internet and the extranet. Redundancy and resiliency are the
main factors for high availability, which requires chassis-based core routers.
• Aggregation layer — The aggregation layer connects with top of rack (ToR) switches and aggregates the traffic into
fewer high-density interfaces such as 10GbE or 40GbE. This layer aggregates the traffic to the core layer.
• Access layer (ToR) — The access layer typically contains ToRs. A ToR is a small form-factor switch that sits on top
of the rack and allows all the servers in the rack to be cabled into the switch. A ToR has a small 1 to 2 rack unit (RU)
form factor.
21
Distributed Core
A distributed core is a two-tier architecture composed of multiple switches interconnected to provide a scalable, highperformance network that replaces the traditional and aggregation layers in a conventional core. Switches are arranged
as spines and leaves; the spines fabric connect the leaves together using a routing protocol. The leaves’ edge ports
connect to the switches, ToR switches, servers, other devices, and the WAN. The spines move traffic between the
leaves bi-directionally, providing redundancy and load balancing. Together, the spine and leaf architecture forms the
distribute core fabric.
This two-tier network design allows traffic to move more efficiently in the core at a higher bandwidth with lower
latencies than most traditional three-tier networks. Because there is no single point of failure that can disrupt the entire
fabric, the distributed core architecture is more resilient and as a result, there is less negative impact on the network
when there is a link or node failure. The AFM views the distributed core as one logical switch.
NOTE: There are no uplinks on the spines. All the leaves have downlinks. The uplink should be configured in the
first two leaves.
Key Advantages
The key advantages of a distributed core architecture are:
• Simplified fabric
• Higher bandwidth
• Highly resilient
• Higher availability
• Low power consumption
• Less cooling
• Lower latency
• Lower cost
• Less rack space
• Easier to scale
22
Distributed Core Terminology
The following terms are unique to the design and deployment of a Layer 3 distributed core fabric.
• Leaf — A switch that connects switches, servers, storage devices, or top-of-rack (TOR) elements. The role of the
leaves switches is to provide access to the fabric. The leaf switch connects to all of spines above it in the fabric.
• Spine — A switch that connects to the leaves switches. The role of the spine is to provide an interconnect to all the
leaves switches. All the ports on the spine switches are used to connect the leaves, various racks together. The
spines provides load balancing and redundancy in the distributed core. There are no uplinks on the spines.
• Edge ports — The uplinks and downlinks on the leaves.
• Uplinks — An edge port link on the first two leaves in the distributed core fabric that connects to the edge WAN,
which typically connects to an internet server provider (ISP).
• Downlinks — An edge port link that connects the leaves to the data access layer; for example, servers or ToR
elements.
NOTE: Specify an even number of uplinks. The minimum number of uplinks is 2. One uplink is for redundancy.
• Fabric Interlinks — Links that connect the spines to the leaves. The fabric interlink bandwidth is fixed: 10 Gb or 40
Gb.
• Fabric over-subscription ratio — Varies the maximum number of available interconnect links. This ratio determines
the number of fabric interlinks (the number of communication links between the spine and leaf devices). The ratio
that you specify depends on the bandwidth, throughput, and edge port requirements. The interlink overoversubscription ratio does not come off the edge port downlinks.
As you increase the fabric over-subscription ratio:
– The total number of ports for the downlinks increases.
– The number of interconnect links from the leaves to the spines decreases.
– The maximum number of available ports increases.
For non-blocking (line rate) between the leaves and spines, select the 1:1 fabric over-subscription ratio. This ratio is
useful when you require a lot of bandwidth and not a lot of ports.
The following image illustrates a distributed core fabric.
23
Important: In a single distributed fabric, all the leaves can act as a non-ToR or as a ToR, not both at the same
time.
Key Considerations for Designing a Distributed Core
When designing the Layer 3 distributed core fabric, consider the following:
• You can deploy up to 10 fabrics. However, the fabrics do not communicate with each other.
• AFM manages Dell S4810, S4820T, S6000, and Z9000 switches.
CAUTION: If you are already using a deployed switch, reset the factory settings. The switch must be in BMP mode.
For information on BMP, see DHCP Integration and the
Z9000 switches at https://www.force10networks.com/CSPortal20/KnowledgeBase/Documentation.aspx. See also
Deployment Topology Use Cases.
The number and type of spines and leaves (switches) in a distributed core fabric are based on the following:
• The type of distributed core fabric design:
– Type 1: Extra Large Core
– Type 2: Large Core
– Type 3: Medium Core
– Type 4: Small Core
• The number of current uplinks and downlinks for the leaves.
• The number of planned edge ports (future uplinks and downlinks) for the leaves.
• Whether you require non-blocking (line rate) performance.
24
FTOS Configuration Guide
for either the S4810, S4820T, S6000, or
• Whether the leaves act as a ToR or are connecting to a server.
• Fabric interlink bandwidth (the links between the spines and leaves).
• Uplinks which are 10 Gb.
• Downlinks which are 1 Gb, 10 Gb, or 40 Gb.
• When the Open Shortest Path First (OSPF) is selected for both uplinks and interlinks, one of the uplinks or interlinks
must be in area 0. If one uplink is in area 0 then the interlinks must not be in area 0.
• The fabric over-subscription ratio.
• Fixed fabric interlink bandwidth that is based on the fabric type: 10 Gb or 40 Gb.
Important: If you do not specify additional links in the fabric design for future expansion in the Bandwidth and Port
Count screen, you can only expand the downlinks on the existing fabric.
For information about how to expand a fabric, see Editing and Expanding an Existing Fabric Design.
Gathering Useful Information for a Distributed Core
To gather the following useful information for a Layer 3 distributed core fabric before you begin:
• Obtain the comma-separated values (CSV) file that contains the system media access control (MAC) addresses,
service tag, and serial numbers for each switch provided from Dell manufacturing or manually enter this information.
• Obtain the location of the switches, including the rack and row number from your network administrator or network
operator.
• Obtain the Remote Trivial File Transfer Protocol (TFTP) or File Transfer Protocol (FTP) address from your network
administrator or network operator. To specify a TFTP/FTP site, go to Administration > Settings > TFTP/FTP screen.
For information about which software packages to use, see the Release Notes.
• Download the software image for each type of switch in the fabric. Each type of switch must use the same version of
the software image within the fabric. Place the software images on the TFTP or FTP site so that the switches can
install the appropriate FTOS software image and configuration file.
• Obtain the Dynamic Host Configuration Protocol (DHCP) server address to be used for the fabric from your DHCP
network administrator or network operator. If a remote DHCP server is not available, AFM also provides a local
DHCP server. The DHCP server must be in the same subnet where the switches are located. After you power cycle
the switches, the switches communicate with the DHCP server to obtain a management IP address based on the
system MAC address. The DHCP server contains information about where to load the correct software image
configuration file for each type of switch from the TFTP/FTP site during BMP. For information about BMP, see DHCP
Integration.
• Obtain pool of IP addresses for the management port for each switch in the fabric.
• Obtain IP addresses (must be an even number) for the uplink configuration from the ISP service. The uplink port
number range is based on whether a 10 Gb or 40 Gb bandwidth is selected.
– For a 10 Gb bandwidth, AFM supports 2 to 32 uplinks.
– For a 40 Gb bandwidth, AFM supports 2 to 8 uplinks.
• Obtain IP addresses for the downlink configuration for connecting to the server or ToR.
• Obtain IP addresses for the fabric link configuration for the spine and leaf switches.
• Gather protocol configuration for uplinks, downlinks and fabric link configuration
25
Selecting a Layer 3 Distributed Core Fabric Design
For large fabric deployments, use the Layer 3 distributed core fabric. AFM supports the following distributed core fabric
designs:
• Type 1: Extra Large Core Fabric
• Type 2: Large Distributed Core Fabric
• Type 3: Medium Distributed Core Fabric
• Type 4: Small Distributed Core Fabric
To select the appropriate Layer 3 distributed core fabric design, use the following table as a guide. For more information
about a Layer 3 distributed core, see:
• Overview of a Distributed Core
• Key Considerations for Designing a Distributed Core Fabric
• Flowchart for Designing and Deploying a Fabric.
With a Layer 3 distributed core topology, you select the Layer 3 option using the Design Wizard on the Deployment
Topology screen. For information about distributed core, see Selecting a Distributed Core Design.
DL BW — Downlink Bandwidth
UL BW — Uplink Bandwidth
Attention: The maximum number of downlinks is based on using 2 uplinks.
Table 1. 2 Tier Layer 3 Distributed Core Topologies
Type OS
Ratio
Type 1-Extra Large
Core
Type 2-Large Core
Type 3-Medium
Core
Type 3-Medium
Core
Type 4-Small Core
Type 4-Small Core
1:1 10G 2046 16
1:1 10G 2046 32
3:1 10G 766 4
4:1 10G 1662 3
5:1 10G 894 2
3:1 10G 1534 4
DL
BW
Maximum #
of
Downlink
Maximum #
of Spine
Devices
Maximum
# of Leaf
Devices
32
64
32
32
8
16
UL BWFabric Link
Bandwidth
Between
the Spine
and Leaf
10G 40G Z9000/Z9000 or
10G 10G S4810/S4810
10G 10G S4810/S4810
10G 40G Z9000/S4810 or
10G 10G S4810/S4810
10G 40G Z9000/S4810 or
Possible
Topologies (Spine
and Leaf)
S6000/S6000
S6000/S4810
S6000/S4810
Type 1: Extra Large Distributed Core Fabric
With a Type 1: Extra Large Distributed Core fabric design, the Z9000 spines (or S6000 spines) connect to the Z9000 leaves
(S6000 leaves) at a fixed 40 Gb line rate. The maximum number of leaves is based on the maximum number of ports on
the spine, 32 ports for the Z9000, as shown in the following figure.
26
Figure 5. Type 1: Extra Large Distributed Core Fabric Design
Use the Type 1: Extra Large Distributed Core fabric design when:
• The line rate-performance with a fabric oversubscription ratio of 1:1 between the spines and leaves.
• The current and future planned uplinks and downlinks on the leaves for the distributed core is less than or equal to
2048 ports.
For redundancy, each leaf in a large core design can connect 2 to 16 spines. The Type 1: Extra Large Distributed Core
Design uses a 1:1 spine-to-leaf ratio. As a result, the maximum number of spines for this design is 16 and the maximum
number of leaves is 32.
Each Z9000 or S6000 leaf for the Type 1: Extra Large Distributed Core design has the following:
• Six hundred forty Gigabit of fabric interlink (fabric links) maximum capacity to the Spine (16 x 40 Gb)
• Forty-eight 10 Gb ports for server connectivity and WAN connectivity
Type 2: Large Distributed Core Fabric
Use the Type 2: Large Distributed Core fabric design when:
• You require a fabric interlink (fabric links) bandwidth between the spines and leaves of 10 Gb is required.
• The current and future planned uplinks and downlinks on the leaves for the fabric is less than or equal to 2048 ports.
• The leaves act as a switch or ToR-leaf switch. Within the ToR, the downlink protocol can be either VLAN or VLAN
and LAG.
With a Type 2: Large Distributed Core fabric design, the S4810 spines connect to the S4810 leaves at a fixed 10 Gb. The
maximum number of spines is 32 and the maximum number of leaves is 64, as shown in the following figure.
27
Figure 6. Type 2: Large Distributed Core Fabric Design
Each S4810 leaf for the Type 2: Large Distributed Core fabric design has the following:
• Forty gigabit of fabric interlink (fabric links) maximum capacity to the spine (4x 10 Gb)
• Thirty-two 10 Gigabit ports will be used for fabric interlink (fabric links) and thirty–two 10 Gb ports are used for the
downlinks
Type 3: Medium Distributed Core Fabric
With a Type 3: Medium Distributed Core design, the Z9000 spines (S6000 spines) connect to the S4810 leaves at a fixed
40 Gb line rate as shown in the following figure. The maximum number of leaves is based on the maximum number of
ports on the spine, 32 ports for the Z9000. The maximum number of spines is 16 and the maximum number of leaves is 32,
as shown in the following illustration. This illustration shows a networking system architecture in a data center are a
distributed core fabric containing a set of ToRs to which servers, storage devices, and network appliances (such as load
balancers or network security appliances) are connected. You can run application services, network services, and
network security services either on physical machines or virtual machines.
28
Figure 7. Type 3: Medium Distributed Core Fabric Design
Use the Type 3: Medium Distributed Core design when:
• You require a fabric interlink (fabric links) bandwidth between the spines and leaves at a 40 Gb line rate.
• The current and future planned uplinks and downlinks on the leaves for your distributed core fabric is less than or
equal to 1536 ports.
• The leaves act as a switch or ToR-leaf switch. Within the ToR, the protocol can be either VLAN or VLAN and LAG.
Each Z9000 spine (S6000 spine) for the Type 3: Medium Distributed Core design has the following:
• Six hundred and forty Gigabit of interlink (fabric links) maximum capacity to the spine (16 x 40 Gig)
• Six hundred and forty 10 Gig Ethernet ports for WAN connectivity
Each S4810 leaf for the Type 3: Medium Distributed Core design has the following:
• One hundred and sixty Gigabit of interlink (fabric links) maximum capacity to the spine (4x 40 Gig)
• Forty–eight 10 Gig Ethernet ports for WAN connectivity
Type 4: Small Distributed Core Fabric
Use the Type 4: Small Distributed Core design when:
• You require a fabric interlink (fabric links) bandwidth between the spines and leaves of 10 Gb.
• The current and future planned uplinks and downlinks on the leaves for your core is less than or equal to 960 ports.
• The maximum port count for a Type 4: Small Distributed Core fabric with an OS ratio of 3:1 is 768. For an OS ratio of
5:1, the maximum port count is 896.
• The leaves act as a switch or ToR-leaf switch. Within the ToR, the downlink protocol can be either VLAN or VLAN
and LAG.
29
With a Type 4: Small Distributed Core fabric design, the S4810 spines connect to the S4810 leaves at a fixed 10 Gb. The
maximum number of spines is 4 and the maximum number of leaves is 16, as show in the following figure.
Figure 8. Type 4: Small Distributed Core Fabric Design
Each S4810 leaf for the Type 4: Small Distributed Core design has the following:
• Sixteen 10 Gigabit of fabric interlink (fabric links) port capacity to the spine
• Forty–eight 10 Gig Ethernet downlinks
• Sixty 10 Gig Ethernet ports for servers per node and WAN connectivity
VLT
Virtual link trunking (VLT) allows physical links between two chassis to appear as a single virtual link to the network core
or other switches such as Edge, Access or Top of Rack (ToR). VLT reduces the role of Spanning Tree protocols by
allowing LAG terminations on two separate distribution or core switches, and by supporting a loop free topology. (A
Spanning Tree protocol is needed to prevent the initial loop that may occur prior to VLT being established. After VLT is
established, RSTP may be used to prevent loops from forming with new links that are incorrectly connected and outside
the VLT domain.) VLT provides Layer 2 multipathing, creating redundancy through increased bandwidth, enabling
multiple parallel paths between nodes and load-balancing traffic where alternative paths exist.
For information about VLT, see the FTOS Configuration Guide for either the S4810, S6000, or the Z9000 at https://
www.force10networks.com/CSPortal20/KnowledgeBase/Documentation.aspx. For more information about VLT, see
Selecting a Layer 2 and Layer 3 with Resiliency (Routed VLT) Fabric Design.
Virtual link trunking offers the following benefits:
• Allows a single device to use a LAG across two upstream devices
• Eliminates Spanning Tree protocol (STP) - blocked ports
• Provides a loop-free topology
• Uses all available uplink bandwidth
30
Loading...