IBM L2-3 User Manual

IBM

Front cover

Nortel Networks L2/3 Ethernet Switch Module for

IBM E

Full Layer 2 switching and Layer 3 routing

Six external multimode fiber or copper GbE interfaces

Hot pluggable switch modules

server BladeCenter

ibm.com/redbooks

Rufus Credle

Stephan Fleck

Scott Lorditch

Jeremy Oliver

Redpaper

International Technical Support Organization

Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter

September 2005

Note: Before using this information and the product it supports, read the information in “Notices” on page vii.

First Edition (September 2005)

This edition applies to Nortel Networks Layer 2/3 Copper and Fiber GbE Switch Modules for IBM Eserver BladeCenter.

Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii

Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

The team that wrote this Redpaper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

Become a published author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .x

Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi

Chapter 1. Executive summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Chapter 2. IBM Eserver BladeCenter overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2.1 The IBM Eserver BladeCenter product family . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.1.1 IBM Eserver BladeCenter storage solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.1.2 IBM Eserver BladeCenter system management . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.2 IBM Eserver BladeCenter architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.2.1 The midplane . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.2.2 Management Module Ethernet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.2.3 Gigabit Ethernet path . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.3 IBM Eserver HS20 architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.4 Stand-alone configuration tools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Chapter 3. Nortel Networks Layer 2/3 GbE Switch Modules. . . . . . . . . . . . . . . . . . . . . 15

3.1 Product description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.2 Value proposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.3 Supported hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Chapter 4. Nortel Networks Layer 2/3 GbE Switch Module architecture . . . . . . . . . . . 21

4.1 Nortel GbESM architecture overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

4.2 Nortel Networks L2/3 GbESM block diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

4.2.1 Nortel Networks L2/3 GbESM ports specific roles . . . . . . . . . . . . . . . . . . . . . . . . 25

Chapter 5. Nortel Networks L2/3 GbESM management and administration . . . . . . . . 29

5.1 Nortel Networks L2/3 GbESM management connectivity . . . . . . . . . . . . . . . . . . . . . . . 30

5.1.1 Out-of-band management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

5.1.2 In-band management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

5.2 Nortel Networks L2/3 GbESM user interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

5.2.1 IBM Eserver BladeCenter Management Module and I2C . . . . . . . . . . . . . . . . . 34

5.2.2 Command-line interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

5.2.3 Browser Based Interface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

5.2.4 SNMP management - IBM Director. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

5.3 Multiple Nortel Networks L2/3 GbESMs in a BladeCenter . . . . . . . . . . . . . . . . . . . . . . 42

Chapter 6. IBM Eserver BladeCenter system initial setup. . . . . . . . . . . . . . . . . . . . . . 43

6.1 IBM Eserver BladeCenter system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

6.1.1 Management Module firmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

6.1.2 Management Module network interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

6.1.3 I/O module management tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

6.2 Blade server initial configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

6.2.1 Firmware update . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

6.2.2 Operating systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

6.2.3 Broadcom Advanced Control Suite installation . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

6.3 Firmware and device drivers used in this example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

Chapter 7. Nortel Networks L2/3 GbESM configuration and network integration . . . 57

7.1 Standards and technologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

7.1.1 VLAN tagging - 802.1Q . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

7.1.2 Link Aggregation and LACP - 802.3ad and 802.3-2002 . . . . . . . . . . . . . . . . . . . . 58

7.1.3 Spanning Tree - 802.1D, 802.1w, 802.1s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

7.1.4 Routing Information Protocol - RFC1058 and RFC2453. . . . . . . . . . . . . . . . . . . . 59

7.1.5 Open Shortest Path First (OSPF) - RFC1257, RFC2328, and others . . . . . . . . . 59

7.1.6 Virtual Router Redundancy Protocol (VRRP) - RFC 3768 . . . . . . . . . . . . . . . . . . 60

7.1.7 Where standards originate and how to get them . . . . . . . . . . . . . . . . . . . . . . . . . 60

7.2 Summary of sample configurations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

7.2.1 Basic Layer 2 configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

7.2.2 Advanced Layer 2 configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

7.2.3 Layer 3 configuration - static routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

7.2.4 Layer 3 configurations - dynamic routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

7.3 Introduction to High Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

7.3.1 Introduction to trunk failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

7.3.2 Introduction to NIC Teaming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

7.3.3 Introduction to VRRP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

7.3.4 Some important rules for ensuring High Availability . . . . . . . . . . . . . . . . . . . . . . . 65

7.4 Guidelines for attaching the BladeCenter to a network . . . . . . . . . . . . . . . . . . . . . . . . . 65

7.4.1 Guidelines and comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

7.5 Base configurations for examples in this document . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

7.5.1 Hardware and software used for lab environment . . . . . . . . . . . . . . . . . . . . . . . . 66

7.5.2 Preconfiguration preparation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

7.5.3 Base configuration common to all examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

7.6 Basic Layer 2 entry topology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

7.6.1 Layer 2 configuration with 802.1Q tagging and trunk failover. . . . . . . . . . . . . . . . 70

7.6.2 Basic topology conclusions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

7.7 Advanced Layer 2 topology sample configurations. . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

7.7.1 Dynamic link aggregation IEEE 802.3ad (LACP) . . . . . . . . . . . . . . . . . . . . . . . . . 82

7.7.2 Common Spanning Tree configuration - IEEE 802.1D and PVST . . . . . . . . . . . . 84

7.7.3 Rapid Spanning Tree IEEE 802.1w . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

7.7.4 Multi-Spanning Tree IEEE 802.1s. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

7.8 Layer 3 topology sample configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

7.8.1 Layer 3 sample configuration with static routing and VRRP . . . . . . . . . . . . . . . . 109

7.8.2 Dynamic routing options OSPF/RIP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

7.9 Configuration for Extreme switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143

Chapter 8. Serial over LAN feature description and configuration . . . . . . . . . . . . . . 151

8.1 SOL overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152

8.2 General rules to establish an SOL connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152

8.3 Configuring SOL for use with the Nortel GbESM . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153

8.4 SOL use during Nortel Networks L2/3 GbESM experiments. . . . . . . . . . . . . . . . . . . . 153

Chapter 9. Nortel Networks Layer 2/3 GbE Switch Module troubleshooting. . . . . . . 155

9.1 Basic rules and unique symptoms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156

9.1.1 Basic rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156

9.2 Nortel Networks L2/3 GbESM troubleshooting methodology . . . . . . . . . . . . . . . . . . . 157

9.2.1 General comments on troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157

9.3 Systematic approach. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158

9.3.1 Problem definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158

iv Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter

9.3.2 Data collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159

9.3.3 Data analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160

9.3.4 Action plan creation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160

9.3.5 Action plan implementation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160

9.3.6 Observation of results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160

9.3.7 Problem resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161

9.4 Troubleshooting tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161

Chapter 10. Service and support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165

10.1 Placing a call to IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166

10.2 Online services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166

10.3 Ordering information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166

10.4 Other support sites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167

Abbreviations and acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169

Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171

IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171

Other publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171

Online resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171

How to get IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173

Help from IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173

Contents v

vi Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter

Notices

This information was developed for products and services offered in the U.S.A.

IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to:

IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A.

The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS

PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.

IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you.

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.

This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental.

COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrates programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and distribute these sample programs in any form without payment to IBM for the purposes of developing, using, marketing, or distributing application programs conforming to IBM's application programming interfaces.

Trademarks

The following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both:

Eserver®

Redbooks (logo) ™ Redbooks (logo)™ eServer™ xSeries® AIX® BladeCenter®

The following terms are trademarks of other companies:

Java, Sun, and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

Microsoft, Windows, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

Intel, Intel logo, Intel Inside logo, and Intel Centrino logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

Domino® Electronic Service Agent™ Enterprise Storage Server® HelpCenter® HelpWare® IntelliStation® IBM® PartnerLink®

Redbooks™ ServerGuide™ Summit® Tivoli® TotalStorage® WebSphere®

Nortel Networks, the Nortel Networks logo, and the globemark design, and Alteon are trademarks of Nortel Networks.

The Extreme Networks logo, Alpine logo, BlackDiamond logo, Summit logos, and Extreme Turbodrive logo are trademarks of Extreme Networks.

Cisco, Cisco IOS, Cisco Systems, the Cisco Systems logo, EtherChannel are Registered trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and certain other countries.

Other company, product, and service names may be trademarks or service marks of others.

viii Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter

Preface

This IBM® Redpaper positions the Nortel Networks Layer 2/3 Fiber and Copper GbE Switch Modules for IBM Eserver BladeCenter and describes how its integrated switch options enable the consolidation of full Layer 2-3 LAN switching and routing capabilities. The Nortel Networks switch modules also provide an upgrade path to full Layer 4-7 services by including 4-7 switch intelligence.

This Redpaper serves as a Best Practices guide for implementing, configuring, and managing Nortel Networks Layer 2/3 Fiber and Copper GbE Switch Modules for several network topologies. Our topology examples include Nortel Networks, Cisco Systems, and Extreme Networks network environments.

This Redpaper can help you to understand the Nortel Networks Layer 2/3 Fiber and Copper GbE Switch Modules architecture. It demonstrates how to use specific tools to manage and administer switch module tasks. It also discusses the differences between Nortel Networks and Cisco Systems terminology.

The audience for this Redpaper is experienced systems and network administrators who want to integrate the Nortel Networks Layer 2/3 Fiber and Copper GbE Switch Modules successfully into new and existing networks.

The team that wrote this Redpaper

This Redpaper was produced by a team of specialists from around the world working at the International Technical Support Organization (ITSO), Raleigh Center.

Rufus Credle is a Certified Consulting I/T Specialist and certified Professional Server Specialist at the ITSO, Raleigh Center. He conducts residencies and develops IBM Redbooks™ and Redpapers that discuss network operating systems, ERP solutions, voice technology, high availability and clustering solutions, Web application servers, pervasive computing, and IBM and OEM e-business applications, all running on IBM Eserver xSeries® and IBM Eserver BladeCenter® technology. Rufus’s various positions during his IBM career have included assignments in administration and asset management, systems engineering, sales and marketing, and IT services. He holds a BS degree in business management from Saint Augustine’s College. Rufus has been employed at IBM for 25 years.

Stephan Fleck is an IBM Accredited Senior IT Specialist at the EMEA ITS/TSS Networking Support Center. He has 12 years of experience in the networking area. Today, he provides EMEA-wide pre- and post-sales support. In addition to his technical skill, Stephan's expertise include project- and critsit-management. During his career, he has been active in product management, deploying new services for the field support group. Stephan is a Cisco Certified Internetwork Expert (CCIE #8301), and he holds a degree in Electrical Engineering from the Technical University Darmstadt, Germany. He has been employed at IBM for 11 years.

Scott Lorditch is a Sales Network Architect for the Blade Switching Server business unit of Nortel Networks. He develops designs and proposals for customers and potential customers of the Nortel Networks GbESM products for the IBM Eserver BladeCenter, including overall network architecture assessments. He also has developed several training and lab sessions for IBM technical and sales personnel and has provided field feedback to the product team. His background before working for Nortel includes almost 20 years working on networking, including electronic securities transfer projects for a major bank based in New York City, as

Senior Network Architect for a multi-national soft drink company, and as Product Manager for managed hosting services for a large telecommunications provider. He holds a BS in Operations Research with specialization in Computer Science from Cornell University.

Jeremy Oliver is a Staff Engineer with the System Validation and Storage group of xSeries Development. He has worked at IBM in Research Triangle Park, North Carolina for seven years. Jeremy's areas of expertise include developing experiments for testing new BladeCenter technologies as well as designing network and power infrastructures to handle test capacity, 10 Gb Ethernet, and operating systems. Jeremy holds a BS degree in Electrical Engineering from McNeese State University, Lake Charles, Louisiana. He also holds a MS degree in Electrical Engineering from North Carolina State University, Raleigh, with research in PHY technologies of computer networking.

Thanks to the following people for their contributions to this project:

Tamikia Barrows, Jeanne Tucker, Margaret Ticknor, ITSO, Raleigh Center

Ishan Sehgal, BladeCenter Marketing Manager, Networking IBM RTP

Paul Woodruff, General Manager of the Blade Server Switching business unit Nortel Networks Santa Clara, CA

Shailesh Naik, Worldwide Director of the Sales Network Architect team Nortel Networks Santa Clara, CA

Mark Davies, IBM Sales & Distribution xSeries FTSS IBM Bermuda

Become a published author

Join us for a two- to six-week residency program! Help write an IBM Redbook dealing with specific products or solutions, while getting hands-on experience with leading-edge technologies. You'll team with IBM technical professionals, Business Partners, or customers.

Your efforts will help increase product acceptance and customer satisfaction. As a bonus, you'll develop a network of contacts in IBM development labs and increase your productivity and marketability.

Find out more about the residency program, browse the residency index, and apply online at:

ibm.com/redbooks/residencies.html

x Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter

Comments welcome

Your comments are important to us!

We want our papers to be as helpful as possible. Send us your comments about this Redpaper or other Redbooks in one of the following ways:

򐂰 Use the online Contact us review redbook form found at:

ibm.com/redbooks

򐂰 Send your comments in an email to:

redbook@us.ibm.com

򐂰 Mail your comments to:

IBM Corporation, International Technical Support Organization Dept. HQ7 Building 662 P.O. Box 12195 Research Triangle Park, NC 27709-2195

Preface xi

xii Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter

Chapter 1. Executive summary

IBM and Nortel Networks are committed to collaborating on the design and development of server and networking technology to address customer requirements by establishing a joint development center. The Nortel Networks Layer 2/3 Copper and Fiber GbE Switch Modules for IBM Eserver BladeCenter (Nortel Networks L2/3 GbESM and Nortel GbESM) represents a new height in this alliance.

The BladeCenter switch module offers BladeCenter customers Nortel’s latest fiber and copper Gigabit Ethernet switching technology which is integrated into the BladeCenter chassis. It further enhances the BladeCenter value proposition by seamlessly interfacing to a customer’s existing data network using six external multimode fiber or copper GbE interfaces.

When installed in the BladeCenter chassis, Nortel Networks L2/3 GbESM provides both full L2 switching and L3 routing capabilities and significant added value not found in commodity switching solutions. This value includes:

򐂰 VLAN tagging - 802.1Q 򐂰 Link Aggregation and LACP - 802.3ad and 802.3-2002 򐂰 Spanning Tree - 802.1D, 802.1w, 802.1s 򐂰 Routing Information Protocol - RFC1058 and RFC2453 򐂰 Open Shortest Path First (OSPF) - RFC1257, RFC2328, and others 򐂰 Virtual Router Redundancy Protocol (VRRP) - RFC 3768

Each Nortel Networks L2/3 GbESM provides one Gigabit per second Ethernet (GbE) connectivity to each of the 14 blade slots and six GbE uplink interfaces external to the IBM Eserver BladeCenter. The customer can install as few as one Nortel Networks L2/3 GbESM or as many as four Nortel Networks L2/3 GbESMs in one BladeCenter chassis. With four Nortel Networks L2/3 GbESMs installed, you can obtain 24 GbE uplink interfaces as well as 56 GbE internal switching capability. The flexibility of the Nortel Networks L2/3 GbESM allows you to address a variety of performance and redundancy needs.

The Nortel and IBM agreement to form a joint development center equips Nortel as it becomes an on demand company that can generate customized products for its network equipment marketplace. This ensures that your needs of high availability, scalability, security, and manageability are addressed. Combined with the integration of IBM Tivoli®, Nortel, and Cisco management products, these architectures bring higher value solutions with lower operational expense.

The Nortel Networks Layer 2/3 Copper and Fiber GbE Switch Modules for IBM Eserver BladeCenter is an integral part of these solutions. With the Nortel Networks L2/3 GbESM, you have the investment protection and price performance of a solution behind which the world’s leading server and networking companies stand.

2 Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter

Chapter 2. IBM Eserver BladeCenter

overview

IBM designed the IBM Eserver BladeCenter innovative modular technology, leadership density, and availability to help solve a multitude of real-world issues.

For organizations seeking server consolidation, the IBM Eserver BladeCenter centralizes servers for increased flexibility, ease of maintenance, reduced cost, and streamlined human resources. Companies that need to deploy new e-commerce and e-business applications can achieve speed while ensuring flexibility, scalability, and availability. For enterprise requirements such as file-and-print and collaboration, the IBM Eserver BladeCenter is designed to offer reliability, flexibility for growth, and cost effectiveness. In addition, clients with compute-intensive applications that need highly available clustering can use the IBM Eserver BladeCenter to help achieve high degrees of scalability and performance.

This chapter provides a high-level overview of the IBM Eserver BladeCenter product family.

2.1 The IBM Eserver BladeCenter product family

The IBM Eserver BladeCenter family of products features a modular design that integrates multiple computing resources into a cost-effective, high-density enclosure for a platform that:

򐂰 Reduces installation, deployment, and redeployment time 򐂰 Reduces administrative costs with our helpful management tools 򐂰 Achieves the highest levels of availability and reliability 򐂰 Provides XpandonDemand scale-out capability 򐂰 Reduces space and cooling requirements compared to 1U solutions

To understand more about how the Nortel Networks Layer 2/3 GbE Switch Module is designed to operate in the BladeCenter chassis, we suggest that you read the sections that follow which discuss the BladeCenter architecture. If you seek to know more about the IBM Eserver BladeCenter and its components, visit:

http://www.ibm.com/products/us/

Figure 2-1 on page 5 shows the IBM Eserver BladeCenter chassis, HS40, HS20, JS20, and LS20:

򐂰 IBM Eserver BladeCenter chassis

The BladeCenter is a high-density blade solution that provides maximum performance, availability, and manageability for application serving, storage flexibility, and long-life investment protection.

򐂰 HS40

HS40 is a 4-way blade server for high-performance enterprise applications requiring four-processor SMP capability. The BladeCenter chassis supports up to seven 4-way servers and is ideal for Enterprise Resource Planning (ERP) and database applications.

򐂰 HS20

The IBM efficient 2-way blade server design offers high density without sacrificing server performance. Ideal for Domino®, Web server, Microsoft® Exchange, file and print, application server, and so on.

򐂰 JS20

JS20 is a 2-way blade server for applications requiring 64-bit computing. Ideal for compute-intensive applications and transactional Web serving.

򐂰 LS20

LS20 is a 2-way blade server running AMD Opteron processors. The LS20 delivers density without sacrificing processor performance or availability. For applications that are limited by memory performance, the LS20 might bring sizeable performance gains.

4 Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter

IBM Eserver BladeCenter

IBM Eserver LS20 IBM Eserver HS20 IBM Eserver HS40

Figure 2-1 IBM Eserver BladeCenter and blade modules

Blade development is ongoing for the BladeCenter platform. Therefore, we suggest that you regularly visit the following Web site for the latest information about IBM Eserver BladeCenter:

http://www.ibm.com/servers/eserver/bladecenter/index.html

2.1.1 IBM Eserver BladeCenter storage solutions

IBM delivers a wide range of easy-to-install, high-capacity, tested storage products for the IBM Eserver BladeCenter to meet your demanding business needs. This enables you to choose from the array of IBM TotalStorage® storage solution products, which include:

򐂰 Fibre Channel products and Storage Area Networks 򐂰 Network Attached Storage 򐂰 Enterprise Storage Server®

IBM TotalStorage provides connected, protected, and complete storage solutions that are designed for your specific requirements, helping to make your storage environment easier to manage, helping to lower costs, and providing business efficiency and business continuity.

For more information about BladeCenter storage solutions, visit:

http://www.pc.ibm.com/us/eserver/xseries/storage.html

IBM Eserver JS20

Chapter 2. IBM Eserver BladeCenter overview 5

2.1.2 IBM Eserver BladeCenter system management

To get the most value from your IBM Eserver BladeCenter investment throughout its life cycle, you need smart, effective systems management which will keep your availability high and costs low.

Management foundation

IBM Director, our acclaimed industry standards-based workgroup software, delivers comprehensive management capability for IntelliStation®, ThinkCentre, ThinkPad, and IBM Eserver BladeCenter and xSeries hardware to help reduce costs and improve productivity. IBM Director is hardware that is designed for intelligent systems management. It offers the best tools in the industry and can save you time and money by increasing availability, tracking assets, optimizing performance, and enabling remote maintenance.

Advanced server management

This exclusive collection of software utilities provides advanced server management and maximum availability through the following components:

򐂰 Server Plus Pack 򐂰 Application Workload Manager 򐂰 Scalable Systems Manager 򐂰 Real-Time Diagnostics 򐂰 Electronic Service Agent™ 򐂰 Tape Drive Management Assistant

For more information about advanced server management, see:

http://www-1.ibm.com/servers/eserver/xseries/systems_management/xseries_sm.html

Deployment and update management

IBM deployment tools help minimize the tedious work that can be involved in getting your servers and clients ready to run. These tools include:

򐂰 Remote Deployment Manager 򐂰 Software Distribution Premium Edition 򐂰 ServerGuide™ 򐂰 ServerGuide Scripting Toolkit 򐂰 UpdateXpress

For more information about IBM Eserver BladeCenter deployment and update management, visit:

http://www.ibm.com/servers/eserver/xseries/systems_management/xseries_sm.html

2.2 IBM Eserver BladeCenter architecture

In this section, we look into the architectural design of the IBM Eserver BladeCenter chassis and its components.

2.2.1 The midplane

Figure 2-2 on page 7 illustrates the BladeCenter midplane. The midplane has two similar sections (upper and lower) that provide redundant functionality. The processor blades (blade servers) plug into the front of the midplane. All other major components plug into the rear of the midplane (for example, power modules, switch modules, and management modules). The processor blades have two connectors, one that is connected to the upper section and one

6 Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter

that is connected to the lower section of the midplane. All other components plug into one section only (upper or lower). However, there is another matching component that can plug into the other midplane section for redundancy.

IBM

BladeCenter™ - Midplane

Front

Panel/Media

Tray

Management

Module 1

Management

Module 2

Switch

Module

CPU

Blade

Switch

Module

Switch

Module

Power

Module

Midplane Upper Section

CPU

Blade

Midplane Lower Section

Switch

Module

Power

Module

Blower

Power

Module

CPU

Blade

Power

Module

Figure 2-2 Midplane view

It should be noted that the upper and lower midplane sections in an IBM Eserver BladeCenter are independent of each other (see Figure 2-3). Having a dual midplane ensures that there is no single point of failure and the blades remain operational.

Figure 2-3 Internal picture of the upper and lower midplane of the BladeCenter chassis

Chapter 2. IBM Eserver BladeCenter overview 7

2.2.2 Management Module Ethernet

Figure 2-4 illustrates the Management Module Ethernet interface. The switch modules are configured by the active Management Module through the use of a 100 Mb Ethernet interface. Each Management Module has four 100 Mb Ethernet interfaces, one for each switch module. Each switch module has two 100 Mb Ethernet interfaces, one for each Management Module.

Note: On the Nortel Networks L2/3 GbESM, the management Ethernet ports on the switch are referred to as MGT1 and MGT2. For more information beyond this generic illustration, see Chapter 4, “Nortel Networks Layer 2/3 GbE Switch Module architecture” on page 21.

The following list clarifies the routing:

򐂰 Management Module 1 Ethernet 1 → Switch Module 1 Ethernet MGT1 򐂰 Management Module 1 Ethernet 2 → Switch Module 2 Ethernet MGT1 򐂰 Management Module 1 Ethernet 3 → Expansion Switch Module 3 Ethernet MGT1 򐂰 Management Module 1 Ethernet 4 → Expansion Switch Module 4 Ethernet MGT1 򐂰 Management Module 2 Ethernet 1 → Switch Module 1 Ethernet MGT2 򐂰 Management Module 2 Ethernet 2 → Switch Module 2 Ethernet MGT2 򐂰 Management Module 2 Ethernet 3 → Expansion Switch Module 3 Ethernet MGT2 򐂰 Management Module 2 Ethernet 4 → Expansion Switch Module 4 Ethernet MGT2

IBM

BladeCenter™ -

Management Module Ethernet Interface

Management

Module 1

Management

Module 2

Switch

Module

CPU

Blade

Switch

Module

Switch

Module

CPU

Blade

Switch Module

Power

Module

Power

Module

Blower

Power

Module

CPU

Blade

Power

Module

Panel/Media

Figure 2-4 Management Module Ethernet interface

The redundant paths of the Management Module Ethernet interface are run from Management Module 2.

Front

Tray

8 Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter

2.2.3 Gigabit Ethernet path

Figure 2-5 on page 10 illustrates the Gigabit Ethernet path. Each processor blade has a minimum of two and a maximum of four EtherLAN interfaces. In particular, the BladeCenter HS20 processor blade has two serializer/deserializer SERDES-based Gb Ethernet interfaces, one for each midplane connector. With a daughter card installed, two more network interfaces can be added. Each switch module (SW Module) receives one LAN input from each processor blade, for a total of 14 inputs.

Note: On the Nortel Networks L2/3 GbESM, the internal Ethernet ports on the switch are referred to as MGT1 and MGT2. For more information beyond this generic illustration, see Chapter 4, “Nortel Networks Layer 2/3 GbE Switch Module architecture” on page 21.

The following partial listing illustrates the routing:

򐂰 Processor blade 1 LAN 1 → Switch Module 1 input INT1 򐂰 Processor blade 1 LAN 2 → Switch Module 2 input INT1 򐂰 Processor blade 1 LAN 3 → Expansion Switch Module 3 input INT1 򐂰 Processor blade 1 LAN 4 → Expansion Switch Module 4 input INT1 򐂰 Processor blade 2 LAN 1 → Switch Module 1 input INT2 򐂰 Processor blade 2 LAN 2 → Switch Module 2 input INT2 򐂰 Processor blade 2 LAN 3 → Expansion Switch Module 3 input INT2 򐂰 Processor blade 2 LAN 4 → Expansion Switch Module 4 input INT2

On processor blade, LAN 1 and LAN 2 are the on-board SERDES Gbit Ethernet interfaces, and are routed to Switch Module 1 and Switch Module 2, respectively, for every processor blade. LAN 3 and LAN 4 go to the Expansion Switch Modules 3 and 4, respectively, and are only to be used when a daughter card is installed. Unless a daughter card is installed in one or more processor blades, there is no need for Switch Modules 3 and 4. Further, the switch modules have to be compatible with the LAN interface generated by the processor blade. If a Fibre Channel daughter card is installed in a BladeCenter HS20 processor blade, Switch Modules 3 and 4 must also be Fibre Channel-based, and any daughter cards installed in the remaining BladeCenter HS20 processor blades must be Fibre Channel.

Chapter 2. IBM Eserver BladeCenter overview 9

IBM

BladeCenter™ -

Gigabit Ethernet path

SERDES

Ethernet

Network

Interface

Daughter

Card

SERDES

Ethernet

Processor blade

Figure 2-5 Gigabit Ethernet path

LAN 1

LAN 3

LAN 4

LAN 2

SW Module 1

1………..14

SW Module 3

1………..14

Midplane (Upper Section)

Midplane (Lower Section)

1………..14

SW Module 2

1………..14

SW Module 4

2.3 IBM Eserver HS20 architecture

In this section, we discuss the architectural design of the IBM Eserver BladeCenter HS20. This is presented as just one example of the blade design for a typical dual-processor server.

10 Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter

The BladeCenter HS20 uses the Intel® Lindenhurst chipset (see the HS20 architecture in Figure 2-6).

8843 HS20 Block Diagram

Due to space limitations this diagram is not drawn to scale

ICH-S

PCI

bus 0

PCIX 66

LSI 1020 SCSI

VRM 10.1

To SP I2C bus

Hublink 1.5

LPC

PCI

32/33

PCI Express x4

PXH

Pri Sec

PCIX B

Daughter card connector

7000M

ATI

Nocona

XEON

CPU

MCH

Data A

Data B

USB ports to HD connectors

VPD 32KB

EEPROM

PCIExpress x8

PCIX C

Video

Servicing the IBM ^ HS20 (M/T 8843) and Blade Storage Expansion-II Option

Nocona

XEON

CPU

To SP I2C bus

DDR2 400Mhz 2GB

To SP

C bus

I2C bus

Video

Renassas SP (2166)

1Gb Ethernet

DIMMs sockets

Broadcom

5704S Ethernet

controller

VRM 10.1

1 Gb Ethernet

SCSI HDD Connector 1

Figure 2-6 HS20 architecture

The Intel Lindenhurst chipset consists of the following components:

򐂰 Memory and I/O controller (MCH) (North Bridge) 򐂰 PXH-D 򐂰 ICH-S (South Bridge)

The Lindenhurst MCH, Memory and I/O controller provides the interface between the processors, the memory, and the PCI Express busses that interface to the other Intel chips. The Lindenhurst ICH-S (South Bridge) provides the USB interfaces, the local Service Processor interface, the POST/BIOS flash EEPROM interface, and the PCI bus interface for the ATI Radeon Mobility Video controller and LSI 1020 SCSI Host Controller. The PXH interfaces the Broadcom BCM5704S ethernet controller on its secondary bus and the daughter card on its secondary bus. I/O functions on the 8843 include Video, I2C, USB, SCSI, Gigabit Ethernet, and USB (floppy, CD-ROM (DVD), mouse, and keyboard).

The LPC bus is used to connect to the POST/BIOS EEPROM on the 8843. The size of the EEPROM is 4 MB x 8, and it contains primary BIOS, backup BIOS, and blade diagnostics.

Blade Expansion Connector

Blade HD

connector A

Midplane HD

connector A

Blade HD

connector B

Midplane HD

connector B

Chapter 2. IBM Eserver BladeCenter overview 11

PCI Express features include:

򐂰 PCI software compatibility 򐂰 Chip-to-chip, board-to-board implementations 򐂰 Support for end-to-end data integrity 򐂰 Advanced error reporting and handling for fault isolation and system recovery 򐂰 Low-overhead, low-latency data transfers and maximized interconnect efficiency 򐂰 High-bandwidth, low pin-count implementations for optimized performance

2.4 Stand-alone configuration tools

IBM Eserver BladeCenter hardware can be configured using standard software, such as a Web browser and a Telnet client, which are available on all the mainstream operating system platforms. This is possible by exploiting Web and American National Standards Institute (ANSI) interfaces that are embedded in both the management and the Ethernet Switch Modules.

A very comprehensive tool is accessible through the Web interface. This tool contains various configuration submenus, and one of them (I/O Module Tasks) lets you set up the Ethernet Switch Module. Basic settings (such as the Ethernet Switch Module IP address and the enablement of the external ports) are configured by exploiting the I2C bus. An advanced menu allows for the fine tuning of the module, by either opening another window of the Web browser or running a Java™ applet that allows for connectivity to an ANSI interface. (This requires that you have Java 2 V1.4 installed on the management system.) To achieve this, the 10/100 Mb internal link that connects the Management Module and the Ethernet Switch Modules through the BladeCenter backplane are exploited (notice that the internal network interface of the Management Module has a default static IP address of 192.168.70.126).

These more complete tools can also be accessed by pointing your Web browser, Telnet, or SSH client to the IP of the Ethernet Switch Module itself. (The default for a module that is plugged into Rear Bay 1 is 192.168.70.127. However, you can configure Dynamic Host Configuration Protocol (DHCP) based addressing.) Notice that this latter capability requires the management system to connect through the external ports (on the production LAN) of the Ethernet Switch Module and, therefore, might potentially raise concerns about security. That is why you have the capability to disable configuration control through the external ports in the I/O Module Tasks of the Management Module interface.

Figure 2-7 on page 13 illustrates the available stand-alone configuration tools.

12 Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter

Management LAN

Management LAN Internal LAN connection

Internal LAN connection Production LAN

Production LAN

Telnet to MM port to manage switch

DHCP lease or

192.168.70.125

Browser

Any of the four

Any of the six

external ports

Can be

disabled

Command Line

(Telnet)

Higher security

MM external

Ethernet port

BladeCenter™ Drawer

Management Module

Web interfaceWeb interface

Switch Module 1

Default is 192.168.70.127*

Web interfaceWeb interface

ANSI interfaceANSI interface

I2C bus

Always static, default is 192.168.70.126

Internal 10/100Mb Ethernet

(Configuration path only as shown by arrow)

Rear Bay 2If Module is plugged into192.168.70.128*This is

Rear Bay 2If Module is plugged into192.168.70.128*This is Rear Bay 3192.168.70.129

Rear Bay 3192.168.70.129 Rear Bay 4192.168.70.130

Rear Bay 4192.168.70.130

Figure 2-7 Stand-alone configuration tools

Chapter 2. IBM Eserver BladeCenter overview 13

14 Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter

Chapter 3. Nortel Networks Layer 2/3 GbE

Switch Modules

This chapter discusses the Nortel Networks Layer 2/3 Copper and Fiber GbE Switch Modules for IBM Eserver BladeCenter and its set of features and services.

3.1 Product description

The new Nortel Networks Layer 2/3 Copper and Fiber GbE Switch Modules for IBM Eserver BladeCenter serve as a switching and routing fabric for the BladeCenter server chassis. In addition to the Layer 2 switching capabilities, these switches introduce the expanded capabilities of Layer 3 routing. Up to four copper or fiber Gb Ethernet modules can reside in the I/O module bays of the BladeCenter chassis. The modules can be hot-plugged into an IBM Eserver BladeCenter without disrupting normal operations.

The Nortel Networks L2/3 GbESM connects to the server blades via the 14 internal GbE interfaces (server ports) over the BladeCenter midplane. It supplies six external copper or multimode fiber GbE interfaces for outside communication (shown in Figure 3-1). The switch is managed via two internal 100 Mbps ports for communication to the BladeCenter management module. A RS232 serial console management interface is also available.

Figure 3-1 Nortel Networks L2/3 GbESM connections

Full Layer 2 switching and Layer 3 routing provide flexible in-chassis traffic management and security. The Nortel Networks Layer 2/3 Copper and Fiber GbE Switch Modules for IBM Eserver BladeCenter provides full Layer 2 switching with availability capabilities such as advanced spanning tree protocols, Link Aggregation Control, Cisco Etherchannel, and

802.1Q VLANs, application delivery and performance features such as granular QOS (Differentiated Service Code Point 802.1p), Internet Group Management Protocol (IGMP) snooping, and multicast.

In particular, the switch modules support up to 16,384 MAC addresses, 4,096 address resolution protocol (ARP) entries, and up to 2,048 dynamic route entries to ensure a high

16 Nortel Networks L2/3 Ethernet Switch Module for IBM Eserver BladeCenter

+ 160 hidden pages