IBM Switch 9077 User Manual

3 Mb
Loading...

IBM 9077 SP Switch Router: Get Connected to the SP Switch

Hajo Kitzhöfer, Steffen Eisenblätter, Uwe Untermarzoner

International Technical Support Organization

http://www.redbooks.ibm.com

SG24-5157-00

SG24-5157-00

International Technical Support Organization

IBM 9077 SP Switch Router:

Get Connected to the SP Switch

November 1998

Take Note!

Before using this information and the product it supports, be sure to read the general information in Appendix D, “Special Notices” on page 305.

First Edition (November 1998)

This edition applies to PSSP Version 2, Release 4 for use with AIX 4.3.1 and Ascend Embedded/OS Version 1.4.6.4.

Comments may be addressed to:

IBM Corporation, International Technical Support Organization Dept. HYJ Mail Station P099

522 South Road

Poughkeepsie, New York 12601-5400

When you send information to IBM, you grant IBM a non-exclusive right to use or distribute the information in any way it believes appropriate without incurring any obligation to you.

© Copyright International Business Machines Corporation 1998. All rights reserved

Note to U.S Government Users – Documentation related to restricted rights – Use, duplication or disclosure is subject to restrictions set forth in GSA ADP Schedule Contract with IBM Corp.

Contents

Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .ix

Tables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xi

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii The Team That Wrote This Redbook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii Comments Welcome . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv

Part 1. Introducing and Installing the GRF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Chapter 1. Dependent Node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.1 Dependent Node Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.2 Limitations of the Dependent Node. . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Chapter 2. Router Node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.1.1 Motivation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.1.2 Design Objectives. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.1.3 What is a Router. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.1.4 Routing without the GRF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.1.5 Routing with the GRF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.1.6 Overview of Supported Routing Protocols. . . . . . . . . . . . . . . . . . 15 2.1.7 Media Adapters At-a-Glance. . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.1.8 Benefits of the GRF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.1.9 Price Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.2 GRF Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.2.1 IP Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.2.2 Supported Routing Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.2.3 Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.2.4 System Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.3 GRF Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.3.1 GRF Block Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.3.2 GRF Features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.3.3 IP Switch and Control Board. . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 2.3.4 Memory Guidelines for the IP Switch Control Board . . . . . . . . . . 35 2.3.5 Characteristics of GRF Media Cards. . . . . . . . . . . . . . . . . . . . . . 36 2.3.6 SP Switch Router Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 2.3.7 Media Card Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 2.3.8 Other Media Cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 2.3.9 GRF Operating Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 2.4 PSSP Enhancements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

© Copyright IBM Corp. 1998

iii

2.4.1 SDR Enhancements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 2.4.2 New Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 2.4.3 Enhanced Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 2.4.4 Hardware Perspectives. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 2.4.5 SP Extension Node SNMP Manager. . . . . . . . . . . . . . . . . . . . . . 58 2.4.6 Dependent Node MIB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 2.4.7 Coexistence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 2.4.8 Partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

2.5 Planning for the GRF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 2.6 Planning for the Dependent Node. . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 2.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

Part 2. Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

Chapter 3. Installation and Configuration . . . . . . . . . . . . . . . . . . . . . . . 69 3.1 Initial Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 3.2 Pre-Installation Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 3.2.1 Order of Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 3.3 Installing an SP Switch Router Adapter Card . . . . . . . . . . . . . . . . . . . 75 3.3.1 Installation Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 3.3.2 Installing the PCMCIA Spinning Disk . . . . . . . . . . . . . . . . . . . . . 76 3.4 Attaching SP Switch Router Cables . . . . . . . . . . . . . . . . . . . . . . . . . . 79 3.4.1 Ethernet Cable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 3.4.2 SP Switch Cable. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 3.4.3 Procedure for Connecting Cards to the SP Switch . . . . . . . . . . . 80 3.5 Configuration Required on the SP System . . . . . . . . . . . . . . . . . . . . . 81 3.5.1 Determining the Switch Connection for a Dependent Node. . . . . 82 3.5.2 Procedure to Get the Jack Number. . . . . . . . . . . . . . . . . . . . . . . 84 3.6 Multiple Frames for Multiple System Connections . . . . . . . . . . . . . . . 85 3.7 Step-by-Step Media Card Configuration . . . . . . . . . . . . . . . . . . . . . . . 86 3.7.1 Configuration Files and Their Uses . . . . . . . . . . . . . . . . . . . . . . . 86 3.8 Step 1. Check SNMP in the SP Switch Router System . . . . . . . . . . . . 88 3.9 Put SNMP Changes into Effect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 3.10 Step 2. Assign IP Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 3.10.1 Method 1: Use SP SNMP Manager - Recommended . . . . . . . . 89 3.10.2 Method 2: Edit /etc/grifconfig.conf - Optional . . . . . . . . . . . . . . 93 3.10.3 Putting grifconfig.conf Additions into Effect . . . . . . . . . . . . . . . . 95 3.11 Step 3. Change Profile Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 3.12 Step 4. Run dev1config . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 3.13 Step 5. Reset SP Switch Router System to Install Files . . . . . . . . . . 96 3.13.1 Saving Configuration Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 3.14 Verify an SP Switch Router Adapter Card on the Router . . . . . . . . . 97 3.14.1 Verify Media Card Operation Using ping . . . . . . . . . . . . . . . . . . 97

iv IBM 9077 SP Switch Router: Get Connected to the SP Switch

3.14.2 Check Media Card Status Using grcard . . . . . . . . . . . . . . . . . . 98 3.14.3 Reset Media Card Using grreset . . . . . . . . . . . . . . . . . . . . . . . . 99 3.14.4 Using grstat to Display GRF Statistics . . . . . . . . . . . . . . . . . . . 99 3.15 Bringing the SP Switch Router Adapter Card Online with the SP . . 100 3.15.1 Checking Connectivity to the SP System . . . . . . . . . . . . . . . . 101

Chapter 4. Configuration of IP-Forwarding Media Cards . . . . . . . . . . 105 4.1 Ethernet 10/100Base-T Configuration. . . . . . . . . . . . . . . . . . . . . . . . 105 4.1.1 Physical and Logical Interfaces . . . . . . . . . . . . . . . . . . . . . . . . 105 4.1.2 Configuration File and Profile Overview . . . . . . . . . . . . . . . . . . 106 4.1.3 Installing Configurations or Changes . . . . . . . . . . . . . . . . . . . . 107 4.1.4 Assign IP Addresses - grifconfig.conf . . . . . . . . . . . . . . . . . . . . 107 4.1.5 Specify Ethernet Card Parameters . . . . . . . . . . . . . . . . . . . . . . 108 4.1.6 Some maint Commands for the Ethernet Media Cards . . . . . . . 109 4.2 ATM OC-3c Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 4.2.1 Physical and Logical ATM Interfaces . . . . . . . . . . . . . . . . . . . . 110 4.2.2 Installing Configurations or Changes . . . . . . . . . . . . . . . . . . . . 113 4.2.3 Configuration Files and Profiles . . . . . . . . . . . . . . . . . . . . . . . . 113 4.2.4 Assign IP Addresses - grifconfig.conf . . . . . . . . . . . . . . . . . . . . 114 4.2.5 Specify ATM Card Parameters . . . . . . . . . . . . . . . . . . . . . . . . . 115 4.2.6 Configuring PVCs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 4.2.7 Some maint Commands for the ATM OC-3c Media Card . . . . . 116 4.2.8 Using grrt to Display the Route Table . . . . . . . . . . . . . . . . . . . . 118 4.2.9 Using grstat to Display GRF Statistics . . . . . . . . . . . . . . . . . . . 119 4.3 ATM OC-12c Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 4.3.1 Physical and Logical ATM Interfaces . . . . . . . . . . . . . . . . . . . . 119 4.3.2 Installing Configurations or Changes . . . . . . . . . . . . . . . . . . . . 120 4.3.3 Configuration Files and Profiles . . . . . . . . . . . . . . . . . . . . . . . . 120 4.4 FDDI Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 4.4.1 Separate Networks versus Bridging . . . . . . . . . . . . . . . . . . . . . 126 4.4.2 Naming the FDDI Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 4.4.3 Physical Interface Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 4.4.4 GRF Interface Name. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 4.4.5 Configuration Files and Profiles . . . . . . . . . . . . . . . . . . . . . . . . 128 4.4.6 Assign IP Addresses - grifconfig.conf . . . . . . . . . . . . . . . . . . . . 129 4.4.7 Specify FDDI Card Parameters. . . . . . . . . . . . . . . . . . . . . . . . . 130 4.4.8 Installing Configurations or Changes . . . . . . . . . . . . . . . . . . . . 130 4.4.9 Some maint Commands for the FDDI Media Card . . . . . . . . . . 131 4.4.10 Using grrt to Display the Route Table . . . . . . . . . . . . . . . . . . . 132 4.4.11 Using grstat to Display GRF Statistics . . . . . . . . . . . . . . . . . . 133 4.5 HIPPI Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 4.5.1 Introduction to HIPPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 4.5.2 HIPPI Configuration Options . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

v

4.5.3 Physical and Logical Interfaces . . . . . . . . . . . . . . . . . . . . . . . . 139 4.5.4 Configuration Files and Profiles . . . . . . . . . . . . . . . . . . . . . . . . 140 4.5.5 Installing Configurations or Changes . . . . . . . . . . . . . . . . . . . . 141 4.5.6 Some maint Commands for the HIPPI Media Card . . . . . . . . . . 141 4.6 Configuring Bridging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 4.6.1 GRF Bridging Implementation. . . . . . . . . . . . . . . . . . . . . . . . . . 142 4.6.2 Simultaneous Routing and Bridging . . . . . . . . . . . . . . . . . . . . . 143 4.6.3 Configuration Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 4.6.4 Interoperability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 4.6.5 Spanning Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 4.6.6 Bridge Filtering Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 4.6.7 Fragmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 4.6.8 Spamming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 4.6.9 Bridging Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 4.6.10 Management Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 4.6.11 Configuration File and Profile Overview . . . . . . . . . . . . . . . . . 148 4.6.12 Bridging ATM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 4.6.13 Bridging FDDI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 4.6.14 Bridging Ethernet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155

Chapter 5. Single RS/6000 SP and Single SP Switch Router . . . . . . . 157 5.1 Single SP Partition and Single SP Switch Router Adapter Card . . . . 157 5.1.1 SP Switch - Ethernet Connection . . . . . . . . . . . . . . . . . . . . . . . 157 5.1.2 SP Switch - FDDI Connection . . . . . . . . . . . . . . . . . . . . . . . . . . 162 5.1.3 SP Switch - ATM Connection . . . . . . . . . . . . . . . . . . . . . . . . . . 167 5.1.4 SP Switch - FDDI Connection (Distinct FDDI Networks) . . . . . . 174 5.1.5 SP Switch - FDDI Connection in an ADSM Environment. . . . . . 185 5.2 Single SP Partition and Multiple SP Switch Router Adapter Cards . . 187 5.2.1 Configuration of a Dual SP Switch Router Connection . . . . . . . 187 5.2.2 Complex Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 5.2.3 Recovery Procedure for an SP Switch Adapter Card Failure. . . 196 5.3 Multiple SP Partition and Multiple SP Switch Router Adapter Cards . 197

Chapter 6. Multiple RS/6000 SPs and One SP Switch Router . . . . . . 203 6.1 RS/6000 SP Switch - RS/6000 SP Switch Connection . . . . . . . . . . . 203 6.2 Sharing Network Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207

Chapter 7. Multiple RS/6000 SPs and Multiple GRFs . . . . . . . . . . . . . 209 7.1 ATM OC-3c Backbone Connection . . . . . . . . . . . . . . . . . . . . . . . . . . 209 7.1.1 ATM OC-3c Backbone - Using One Port . . . . . . . . . . . . . . . . . . 210 7.1.2 ATM OC-3c Backbone - Using Two Ports . . . . . . . . . . . . . . . . . 215 7.2 ATM OC-12c Backbone - One Port. . . . . . . . . . . . . . . . . . . . . . . . . . 222 7.3 HIPPI Backbone Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227

vi IBM 9077 SP Switch Router: Get Connected to the SP Switch

Appendix A. Laboratory Hardware and Software Configuration . . . . 233 A.1 Node and Control Workstation Configuration . . . . . . . . . . . . . . . . . . . . . 233 A.1.1 Hard Disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235 A.1.2 Software Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239 A.1.3 Network Options and Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253 A.2 SP Switch Pool Size Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255 A.3 7025-F50 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256 A.4 SP IP Switch Router Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258

Appendix B. GRF Configuration Files . . . . . . . . . . . . . . . . . . . . . . . . . . 261 B.1 /root/.profile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261 B.2 /etc/Release . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263 B.3 /etc/bridged.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264 B.4 /etc/fstab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267 B.5 /etc/grarp.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267 B.6 /etc/gratm.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268 B.7 /etc/grclean.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274 B.8 /etc/grclean.logs.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275 B.9 /etc/grdev1.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277 B.10 /etc/grifconfig.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282 B.11 /etc/grlamap.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284 B.12 /etc/grroute.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285 B.13 /etc/hosts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 B.14 /etc/inetd.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 B.15 /etc/motd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287 B.16 /etc/rc.local . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287 B.17 /etc/snmpd.conf. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288 B.18 /etc/syslog.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291 B.19 /etc/ttys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292

Appendix C. Hardware and Software Information . . . . . . . . . . . . . . . . 295 C.1 The Front Panel of the SP Switch Router Adapter Card - Operational. . 295 C.2 SP Switch Router Adapter Media Card LEDs. . . . . . . . . . . . . . . . . . . . . 296 C.3 SP Switch Router Adapter Media Card - Bootup . . . . . . . . . . . . . . . . . . 297 C.4 Connectors and Receptacles for Different Media . . . . . . . . . . . . . . . . . . 298 C.5 Chip Interconnection on the TBS Board . . . . . . . . . . . . . . . . . . . . . . . . . 298 C.6 Updating Router Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299

C.6.1 The SP Switch Router as an IBM Product . . . . . . . . . . . . . . . . . . . 299 C.6.2 Obtaining New Machine Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300 C.6.3 Support for Code Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300 C.6.4 Sample Steps to Upgrade the System Software . . . . . . . . . . . . . . 300 C.6.5 Sample Execution of grf_update Script . . . . . . . . . . . . . . . . . . . . . 301

vii

Appendix D. Special Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305

Appendix E. Related Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309 E.1 International Technical Support Organization Publications . . . . . . . . . . 309 E.2 Redbooks on CD-ROMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309 E.3 Other Publications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309

How to Get ITSO Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311 How IBM Employees Can Get ITSO Redbooks . . . . . . . . . . . . . . . . . . . . . . . 311 How Customers Can Get ITSO Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . 312 IBM Redbook Order Form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313

List of Abbreviations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317

ITSO Redbook Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323

viii IBM 9077 SP Switch Router: Get Connected to the SP Switch

Figures

1. SP Switch Router. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2. Functional Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 3. Typical Router Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 4. Table-Based Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 5. Routing without GRF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 6. Routing with GRF. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 7. GRF 400 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 8. Conventional Routers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 9. Switched Routers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 10. Price Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 11. GRF Models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 12. GRF Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 13. Data Packet Transfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 14. Routing Packet Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 15. Side View of GRF 400 Chassis with Slots Numbered . . . . . . . . . . . . . . . . 32 16. Top View of the GRF 1600 Chassis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 17. IP Switch Control Board. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 18. System RAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 19. SP Switch Router Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 20. Hardware Perspectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 21. Action Menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 22. Hardware Notebook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 23. System Partition Aid Perspectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 24. System Partition Aid Notebook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 25. Coexistence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 26. Partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 27. The Laboratory Hardware Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 28. Connecting the GRF to the SP Switch and the CWS . . . . . . . . . . . . . . . . 69 29. Connecting the GRF to the Frame. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 30. Connecting the GRF Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 31. SP System Administrative Ethernet Connections . . . . . . . . . . . . . . . . . . . 80 32. Switch Port Assignments in Supported Frame Configurations . . . . . . . . . 83 33. Node Numbering for an SP System. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 34. How Frames Enable Connections to Multiple SP Switches. . . . . . . . . . . . 86 35. Components in the SP Switch Router Adapter Card’s Interface Name . . . 93 36. Components of the Ethernet Interface Name . . . . . . . . . . . . . . . . . . . . . 106 37. ATM OC-3c Physical and Logical Interfaces . . . . . . . . . . . . . . . . . . . . . . 110 38. Components in the ATM OC-3c Interface Name . . . . . . . . . . . . . . . . . . . 111 39. Components Forming a Virtual Path . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 40. ATM OC-12c Physical and Logical Interfaces . . . . . . . . . . . . . . . . . . . . . 120

© Copyright IBM Corp. 1998

ix

41. Master/Slave Connectors for SAS Interfaces . . . . . . . . . . . . . . . . . . . . . 122 42. A/B Connectors for DAS Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 43. Allowed SAS and DAS Configurations. . . . . . . . . . . . . . . . . . . . . . . . . . . 123 44. Optical Bypass Switch Attachments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 45. Dual Homing Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 46. Assigning Numbers to FDDI Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . 127 47. Physical Interface Numbering on the FDDI Media Card . . . . . . . . . . . . . 128 48. GRF Interface Name for FDDI Interfaces . . . . . . . . . . . . . . . . . . . . . . . . 128 49. HIPPI I-Field Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 50. Components in the HIPPI Interface Name. . . . . . . . . . . . . . . . . . . . . . . . 139 51. Interface Name for FDDI, Ethernet and ATM OC-3c Interfaces . . . . . . . 150 52. One Card - One SP Partition Sample Configuration . . . . . . . . . . . . . . . . 157 53. SP Switch - Ethernet Connection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 54. SP Switch - FDDI Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 55. SP Switch - ATM Connection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 56. SP Switch - FDDI Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 57. SP Switch - FDDI Connection (Bridging) . . . . . . . . . . . . . . . . . . . . . . . . . 180 58. SP Switch Router in an ADSM Environment . . . . . . . . . . . . . . . . . . . . . . 185 59. Connecting One SP Switch with Two SP Switch Router Adapter Cards . 187 60. Configuration with Dual SP Switch Router - SP Switch Connection . . . . 190 61. IP Traffic Flow When Issuing ping 192.168.13.1 on Node 6 . . . . . . . . . . 195 62. IP Traffic Flow When Issuing ping 192.168.13.1 on Node 10 . . . . . . . . . 195 63. IP Traffic Flow When Issuing ping 192.168.13.1 on Node 8 . . . . . . . . . . 196 64. Partition-to-Partition Connection with an SP Switch Router . . . . . . . . . . 198 65. Two RS/6000 SPs Connected to GRF 1600 . . . . . . . . . . . . . . . . . . . . . . 203 66. Sharing Network Resources between Two SPs . . . . . . . . . . . . . . . . . . . 207 67. Connection of Two SPs with Two SP Switch Routers . . . . . . . . . . . . . . . 209 68. SP Switch - ATM - SP Switch Connection . . . . . . . . . . . . . . . . . . . . . . . . 211 69. SP Switch - ATM Bridged - SP Switch Connection . . . . . . . . . . . . . . . . . 215 70. SP Switch - ATM OC-12c - SP Switch Connection . . . . . . . . . . . . . . . . . 223 71. SP Switch - HIPPI - SP Switch Connection . . . . . . . . . . . . . . . . . . . . . . . 228 72. Front Panel of the SP Switch Router Adapter Card with LEDs . . . . . . . . 295 73. The SP Switch Board. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299

x IBM 9077 SP Switch Router: Get Connected to the SP Switch

Tables

1. Memory Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 2. DependentNode Attributes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 3. DependentAdapter Attributes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 4. Additional SDR Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 5. New Commands (root Executable) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 6. New Commands (User Executable). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 7. endefnode Command Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 8. enrmnode Command Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 9. endefadapter Command Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 10. enadmin Command Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 11. splstnode Command Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 12. splstadapter Command Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 13. Enhanced Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 14. Configuration of SP Switch - Ethernet Connection . . . . . . . . . . . . . . . . . 159 15. Configuration of an SP Switch - FDDI Connection . . . . . . . . . . . . . . . . . 163 16. Configuration of SP Switch - ATM Connection . . . . . . . . . . . . . . . . . . . . 168 17. Configuration of SP Switch - FDDI Connection . . . . . . . . . . . . . . . . . . . . 175 18. Configuration of SP Switch - FDDI Connection (Bridging). . . . . . . . . . . . 181 19. Configuration of a Dual SP Switch Router Connection . . . . . . . . . . . . . . 187 20. Configuration of a Dual SP Switch Router - SP Switch Connection . . . . 191 21. Configuration of a Partition - Partition Connection . . . . . . . . . . . . . . . . . 199 22. Configuration of SP Switch - SP Switch Connection . . . . . . . . . . . . . . . . 204 23. Configuration of SP Switch - ATM - SP Switch . . . . . . . . . . . . . . . . . . . . 212 24. Configuration of SP Switch - ATM Bridged - SP Switch . . . . . . . . . . . . . 216 25. Configuration of SP Switch - ATM OC-12c - SP Switch . . . . . . . . . . . . . 224 26. Configuration of SP Switch - HIPPI - SP Switch . . . . . . . . . . . . . . . . . . . 228 27. Configuration of SP 21. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234 28. Configuration of SP 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235 29. Hard Disk Equipment of SP 21 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236 30. Hard Disk Equipment of SP 2 Part 1 of 2. . . . . . . . . . . . . . . . . . . . . . . . . 237 31. Hard Disk Equipment of SP 2 Part 2 of 2. . . . . . . . . . . . . . . . . . . . . . . . . 238 32. Software Levels on CWS and All Nodes Part 1 of 14 . . . . . . . . . . . . . . . 239 33. Software Levels on CWS and All Nodes Part 2 of 14 . . . . . . . . . . . . . . . 240 34. Software Levels on CWS and All Nodes Part 3 of 14 . . . . . . . . . . . . . . . 241 35. Software Levels on CWS and All Nodes Part 4 of 14 . . . . . . . . . . . . . . . 242 36. Software Levels on CWS and All Nodes Part 5 of 14 . . . . . . . . . . . . . . . 243 37. Software Levels on CWS and All Nodes Part 6 of 14 . . . . . . . . . . . . . . . 244 38. Software Levels on CWS and All Nodes Part 7 of 14 . . . . . . . . . . . . . . . 245 39. Software Levels on CWS and All Nodes Part 8 of 14 . . . . . . . . . . . . . . . 246 40. Software Levels on CWS and All Nodes Part 9 of 14 . . . . . . . . . . . . . . . 247

© Copyright IBM Corp. 1998

xi

41. Software Levels on CWS and All Nodes Part 10 of14 . . . . . . . . . . . . . . . 248 42. Software Levels on CWS and All Nodes Part 11 of 14 . . . . . . . . . . . . . . 249 43. Software Levels on CWS and All Nodes Part 12 of 14 . . . . . . . . . . . . . . 250 44. Software Levels on CWS and All Nodes Part 13 of 14 . . . . . . . . . . . . . . 251 45. Software Levels on CWS and All Nodes Part 14 of 14 . . . . . . . . . . . . . . 252 46. Network Options of CWS and All Nodes Part 1 of 3 . . . . . . . . . . . . . . . . 253 47. Network Options of CWS and All Nodes Part 2 of 3 . . . . . . . . . . . . . . . . 254 48. Network Options of CWS and All Nodes Part 3 of 3 . . . . . . . . . . . . . . . . 255 49. Network Options of 7025-F50 Part 1 of 3 . . . . . . . . . . . . . . . . . . . . . . . . 256 50. Network Options of 7025-F50 Part 2 of 3 . . . . . . . . . . . . . . . . . . . . . . . . 257 51. Network Options of 7025-F50 Part 3 of 3 . . . . . . . . . . . . . . . . . . . . . . . . 258 52. SP Switch Router Adapter Media Card LEDs . . . . . . . . . . . . . . . . . . . . . 296 53. SP Switch Router Adapter Media Card LEDs - RX/TX . . . . . . . . . . . . . . 296 54. SP Switch Router Adapter Media Card LEDs During Bootup . . . . . . . . . 297 55. Media Card Cables and Connectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298

xii IBM 9077 SP Switch Router: Get Connected to the SP Switch

Preface

The GRF is a high-performance switched IP Router which provides high-speed data communication links between IBM RS/6000 SP and external networks or hosts. It acts as a special-purpose SP node that routes IP traffic between SP nodes on the SP Switch and the outside world. Connected directly to the SP Switch, the router offers significantly improved SP I/O performance. When packaged with an IBM SP system, the GRF router is referred to as the SP Switch Router.

This redbook helps you install, tailor and configure the SP Switch Router, IBM machine type 9077. The SP Switch Router is also known as the "Gigarouter" or High Performance Gateway Node (HPGN).

The first part of the book gives an overview of the GRF architecture and how the router was integrated into the SP. It emphasizes the advantages of choosing a dedicated router node in some configurations, as opposed to using standard nodes for the routing task. This part also describes some routing fundamentals, particularly focusing on concepts like IPand switch-routing.

The second part presents sample configurations that were carefully chosen to match frequently occurring customer situations. The basic configurations shown are building blocks for more complex networking topologies that include the SP Switch Router and may inspire more sophisticated configurations. All configurations described were tested and provide some comparable performance figures.

This publication is intended to give IBM customers, system engineers, and marketing personnel a broad understanding of this new architecture and what it is used for.

The Team That Wrote This Redbook

This redbook was produced by a team of specialists from around the world working at the International Technical Support Organization, Poughkeepsie Center.

Dr Hajo Kitzhöfer is an Advisory International Technical Support Organization (ITSO) Specialist for RS/6000 SP at the Poughkeepsie Center. He holds a Ph.D. degree in electrical engineering from the Ruhr-University of Bochum (RUB). Before joining ITSO, he worked as an SP Specialist at the RS/6000 and AIX Competence Center, IBM Germany. He has worked at IBM

© Copyright IBM Corp. 1998

xiii

for eight years. His areas of expertise include RS/6000 SP, SMP, and Benchmarks. He now specializes in SP System Management, SP Performance Tuning and SP hardware.

Dr Steffen Eisenblätter is an AIX Software Specialist in the RS/6000 SP Software Support Center, Germany. He holds a Ph.D. degree in physics from the University of Leipzig. He joined IBM in 1997 and has focused on RS/6000 SP products and TCP/IP.

Uwe Untermarzoner is an RS/6000 SP Technical Support Specialist with IBM Germany. He joined IBM 1989. He has ten years of experience in AIX and five years of experience with the SP, mostly in the commercial environment. He joined IBM at 1989. His areas of expertise include AIX, RS/6000 SP, SMP, PSSP, Networking, Performance Tuning and Systems Management.

Thanks to the following people for their invaluable contributions to this project:

Ronald Linton

IBM PPS Lab Poughkeepsie

Gene Novitsky

Ascend Communications, Inc.

Frank May

IBM Worldwide RS/6000 SP Product Marketing

Wes Kinard

IBM RS/6000 Networking Technologies

Marcelo R. Barrios

International Technical Support Organization, Poughkeepsie Center

xiv IBM 9077 SP Switch Router: Get Connected to the SP Switch

Comments Welcome

Your comments are important to us!

We want our redbooks to be as helpful as possible. Please send us your comments about this or other redbooks in one of the following ways:

Fax the evaluation form found in “ITSO Redbook Evaluation” on page 323 to the fax number shown on the form.

Use the electronic evaluation form found on the Redbooks Web sites:

For Internet usershttp://www.redbooks.ibm.com For IBM Intranet usershttp://w3.itso.ibm.com

Send us a note at the following address:

redbook@us.ibm.com

xv

xvi IBM 9077 SP Switch Router: Get Connected to the SP Switch

Part 1. Introducing and Installing the GRF

© Copyright IBM Corp. 1998

1

2 IBM 9077 SP Switch Router: Get Connected to the SP Switch

Chapter 1. Dependent Node

This chapter provides an overview of a dependent node in RS/6000 SP. We start by defining the dependent node and the rationale behind its design.

1.1 Dependent Node Architecture

The Dependent Node Architecture refers to a processor or node, possibly not provided by IBM, for use with the RS/6000 SP.

Since a dependent node may not be a regular RS/6000 SP node, not all the functions of a node can be performed on it, which is why it is called "dependent". For example, it does not allow all the functions of the fault service (Worm) daemon, as other RS/6000 SP nodes with access to the SP Switch do.

The objective of this architecture is to allow the other processors or hardware to easily work together with the RS/6000 SP, extending the scope and capabilities of the system.

The dependent node connects to the RS/6000 SP Switch (but not to the earlier High Performance Switch, HiPS).

The SP Switch Router Adapter is the first product to exploit the Dependent Node Architecture.

1.2 Limitations of the Dependent Node

The following are limitations associated with use of the dependent node:

To use the dependent node in an RS/6000 SP requires the SP Extension Node SNMP Manager to be installed in the Control Workstation. The SP Extension Node SNMP Manager requires UDP port 162 in the Control Workstation. Other SNMP managers, such as Netview, also require this port. To allow the two SNMP managers to coexist, the SP Extension Node SNMP Manager must use an alternative UDP port.Dependent nodes are not allowed in Node Groups.

Only the 8-port and 16-port SP Switch are supported. The 8-port and 16-port High Performance Switch (the old SP Switch) are not supported.

The spmon command on the RS/6000 SP is not enhanced to support dependent nodes. Dependent nodes can only be viewed with the perspectives command.

© Copyright IBM Corp. 1998

3

The fault service daemon runs on all switch nodes in the RS/6000 SP, but not on the dependent node. Therefore, the dependent node does not have the full functionality of a normal RS/6000 SP Switch node.

The dependent node requires the SP Switch’s primary node to compute its switch routes. Therefore, the primary node must have at least PSSP 2.3 installed, otherwise the dependent node cannot work with the RS/6000 SP.

In the RS/6000 SP, SP Switch nodes occasionally send service packets from one node to the next to keep track of status and links. Sometimes these packets are sent indirectly through another switch node. As the dependent node is not a standard RS/6000 SP Switch node, it cannot be used to forward service packets to other nodes.

4 IBM 9077 SP Switch Router: Get Connected to the SP Switch

Chapter 2. Router Node

The first dependent node is actually a new SP Switch Router Adapter in a router. This chapter offers more details about the implementation.

Section 2.1, “Overview” on page 5 gives you an overview of SP Switch Router. This is probably the best to get an impression what the GRF is good for. Also a functionaland a price-comparison between using an RS/6000 SP node and the SP Switch Router is included.

More details about the underlaying Software and Hardware can be found in Section 2.2, “GRF Software” on page 18 and Section 2.3, “GRF Hardware” on page 24.

Section 2.4, “PSSP Enhancements” on page 40 describes the enhancements in the PSSP Software for the support of the dependent node.

Some planning considerations which should be considered can found in Section 2.5, “Planning for the GRF” on page 63 and Section 2.6, “Planning for the Dependent Node” on page 65.

2.1 Overview

The purpose of the SP Switch Router Adapter is to allow the GRF ("goes really fast"), manufactured by Ascend, to forward SP Switch IP traffic to other networks. The GRF was known as the High Performance Gateway Node (HPGN) during the development of the adapter. IBM remarkets models of the GRF that connect to the SP Switch as the SP Switch Router model 04S (9077-04S) and model 16S (9077-16S). These models are not available directly from Ascend.

Note: In the remainder of this book, we refer to the SP Switch Router as the GRF.

The distinguishing feature of the GRF, when compared with other routers, is that it has an SP Switch Router Adapter and therefore can connect directly to the SP Switch (see Figure 1 on page 6).

© Copyright IBM Corp. 1998

5

IBM 9570 Disk

Array

Subsystem

SP Switch

HIPPI

Adapter

Adapter

HIPPI

Adapter

 

 

ATM

 

 

 

OC-12c

ATM

 

 

ATM

Switch

SP

SP Switch

OC-3c

 

Adapter

 

 

 

 

 

Switch

Processor

 

Nodes

8-port

 

E-net 10/100

SP Switch

Adapter

4-port FDDI

SP Switch

Router

SP System

Figure 1. SP Switch Router

The RS/6000 SP software treats this adapter as an extension node. It is a node because it takes up one port in the SP Switch and is assigned a node number. It is described as an extension because it is not a standard RS/6000 SP node, but an adapter card that extends the scope of the RS/6000 SP.

Although the term extension node represents the node appearance of the adapter, it does not define the connection. An extension node adapter is used for that purpose. Each extension node has an extension node adapter to represent its connection to the SP Switch.

2.1.1 Motivation

A thin node, which has a single microchannel, is unable to deliver more than about 30 MB/s to or from the SP Switch. Using a wide node, this number increases to 65 MB/s but is still unable to provide full bandwidth to even one HIPPI interface. It is also unable to feed 4 FDDI or 4 Ethernet 100BaseTx cards at full bandwidth.

A 135 MHz wide node’s CPU becomes saturated at about 5000 packets/second. A 10 Mb/s Ethernet uses a maximum of 1500 bytes for a

6 IBM 9077 SP Switch Router: Get Connected to the SP Switch

packet size. This would only enable a wide node to handle approximately 7.5 MB/s of IP traffic.

Since Ascend’s business depends on keeping pace with networking technology, they already support the major interfaces today. The 9077 will be able to take advantage of any new interfaces that are developed in the future as well, with no further development time or money expended.

With some interfaces requiring up to 5 slots, even a wide node can run out of available slots. This forces additional nodes to be added even if there are no performance limitations in the current configuration.

Since there are no hot plug capabilities with an SP node, any failure means downtime on all interfaces configured in that node, and at times a lengthy maintenance procedure. Redundancy is not built into the SP node’s architecture.

These facts are illustrated in Figure 2:

 

 

 

SP Node

9077

Bus

Shared

Non-blocking Crosspoint Switch

 

 

 

1 MCA per thin node

250 ns path setup

 

 

 

2 MCA per wide node

 

 

 

Route Table

Centralized

Independent lookups per card

 

Cache,Software Based

Hardware based, <2.5 µs

 

Cache hits <50% typical

150,000 route capacity per card

Scalability

Single port per card

High per card port density

 

Single CPU

Per card

 

Limited by shared bus

 

 

Processors

 

 

 

 

 

 

 

 

 

Route Tables

 

 

 

 

 

 

 

 

 

 

 

 

Lookup engines

 

 

 

 

 

 

 

 

 

 

Each card has dedicated bandwidth

Throughput

5000 pps

Up to 130,000 pps

 

30 MB/s per thin node

100 MB/s per card slot, full duplex

 

65 MB/s per wide node

 

 

 

Support

No support for:

Support for:

 

 

 

HSSI

 

 

multiple SP Switch interfaces

 

 

 

 

 

 

 

 

ATM OC12

 

 

High-speed networks such as

 

 

 

 

 

 

 

 

Sonet

 

 

HIPPI

 

 

 

Multiple SP Switch Adapters

 

 

Protocols

 

 

 

 

 

 

 

Figure 2. Functional Comparison

Router Node 7

2.1.2 Design Objectives

Because the dependent node is part of the RS/6000 SP, it had to be packaged and assigned some roles consistent with other RS/6000 SP nodes. Changes were made to the RS/6000 SP to incorporate management requirements for the dependent node.

Ease of design and implementation were important objectives in the design. These were accomplished by limiting the amount of switchcontrol protocol for the dependent node.

New SDR (System Data Repository) classes were created to manage dependent nodes. This was done to minimize the scope of the changes and the exposure to side effects that dependent nodes may cause if they were represented as standard nodes in the SDR.

2.1.3 What is a Router

One of the basic functions of the Internet Protocol (IP) is its ability to connect between different networks. This is due its routing algorithm and its flexibility to use almost any physical network below. A system that connects different physical or logical networks and directs traffic is termed a router, although the older term IP gateway is also used.

Again, IP routing is the passing of an IP packet from one device to another by sending it on a physical or logical interface. routers interconnect networks so that IP traffic can be routed between the systems in the networks, as shown in Figure 3 on page 9.

8 IBM 9077 SP Switch Router: Get Connected to the SP Switch

Network 1

Network 3

Router

Network n

Network 2

Figure 3. Typical Router Configuration

Routers help to reduce the amount of processing required on local systems, since they perform the computation of routes to remote systems. For example, a system can communicate with a remote system by passing the message (or packets) to the router. The router works out how to get to the remote system and forwards the message appropriately.

Storing routes on the system takes up memory. But because a system does not have to store routes to systems not in its own subnet, the route table uses less storage space and thereby frees up memory for other work.

The use of routing reduces network traffic, because routers encourage subnetting, which creates a smaller network of systems. By having smaller networks, network traffic congestion is reduced and overall network performance and traffic control are improved.

A network’s routing configuration does not always require a routing protocol. In situations where the routing information does not change, for example, when there is only one possible route, the system administrator usually builds the routing table manually. Some networks have no access to any other TCP/IP networks, and therefore do not require routing tables at all. The three most common routing configurations are:

Router Node 9

Minimal routing

A network completely isolated from all other TCP/IP networks requires only minimal routing. A minimal routing table is usually built by ifconfig when the network interfaces are configured. If your network does not have direct access to other TCP/IP networks, and if you are not using subnetting, this may be the only routing table you require.

Static routing

A network with a limited number of gateways to other TCP/IP networks can be configured with static routing. When a network has only one gateway, a static route is the best choice. A static routing table is constructed manually by the system administrator using the route command. See Figure 4. Static routing tables do not adjust to network changes, so they work best where routes do not change.

Source Host

 

 

Destination Host

Application

 

 

Application

Transport

Gateway

Transport

Destination

Gateway

Destination

Gateway

Destination

Gateway

192.168.1.0

192.168.12.3

192.168.1.0

192.168.1.5

192.168.1.0

192.168.1.2

192.168.12.0

192.168.12.2

192.168.12.0

192.168.12.3

default

192.168.1.5

default

192.168.12.1

default

192.168.12.1

 

 

Network Access

Network Access

Network Access

192.168.12.2

192.168.12.3

192.168.1.5

192.168.1.2

 

192.168.12.0

 

192.168.1.0

 

Figure 4. Table-Based Routing

Dynamic routing

A network with more than one possible route to the same destination should use dynamic routing. A dynamic routing table is built from the information exchanged by the routing protocols. The protocols are designed to distribute information that dynamically adjusts routes to reflect changing network conditions. Routing protocols handle complex routing

10 IBM 9077 SP Switch Router: Get Connected to the SP Switch

situations more quickly and accurately than a system administrator can do. Routing protocols are designed not only to switch to a backup route when the primary route becomes inoperable; they are also designed to decide which is the "best" route to a destination. On any network where there are multiple paths to the same destination, a dynamic routing protocol should be used.

2.1.4 Routing without the GRF

Before the GRF was available, there were only two ways to get IP traffic from remote systems to reach the RS/6000 SP nodes:

1.By putting an additional IP adapter into every RS/6000 SP node.

2.By designating one or two nodes to act as a router (as shown in Figure 5).

 

Node

Router

 

Internet/Intranet

 

 

 

Node

 

SP Switch

Node

ATM

 

 

. . .

 

 

Node

FDDI

 

 

 

 

Ethernet

Figure 5. Routing without GRF

The first option was usually not chosen because it was too costly for the following reasons:

For systems with a large number of nodes having multiple IP adapters for each RS/6000 SP node can be expensive.

The number of I/O slots in the RS/6000 SP node is limited. In addition, these slots are required to perform other tasks for the system, such as connecting to disk or tape. Using these I/O slots to connect IP adapters restricts the functions of the RS/6000 SP node.

Router Node 11

The second case has proven to be very expensive as well. The RS/6000 SP node was not designed for routing. It is not a cost-effective way to route traffic for the following reasons:

It takes many CPU cycles to process routing. The CPU is not a dedicated router and is very inefficient when used to route IP traffic (this processing can result in usage of up to 90%).

It takes a lot of memory to store route tables. The memory on the RS/6000 SP node is typically more expensive than router memory.

The CPU on a node can only drive the system I/O bus at less than 80 megabytes per second, which is less than what a high-end router can do.

For these reasons, the performance of routers in handling IP traffic from remote systems to the RS/6000 SP nodes was limited.

2.1.5 Routing with the GRF

The GRF is a dedicated, high-performance router (see Figure 6). Each SP Switch Router adapter can route up to 30,000 packets per second and up to 100 MB/s into the SP Switch network in each direction simultaneously.

Node

 

Internet/Intranet

Node

 

 

. . .

 

 

SP Switch

 

 

Node

 

ATM

 

 

 

GRF

FDDI

 

 

 

 

Ethernet

Figure 6. Routing with GRF

12 IBM 9077 SP Switch Router: Get Connected to the SP Switch

The GRF uses a crosspoint switch (see Figure 7) instead of an I/O bus to interconnect its adapters. This switch is capable of 4 or 16 Gbit/s (model dependent) and gives better performance than the MCA bus.

IP Switch Control Board

 

 

 

Route

 

4Gb/s

Manager

 

Crosspoint

 

 

Switch

 

 

1 Gb/s to each Media Card

 

Switch Engine Interface

T3-OC12

 

 

 

Route

I/O

IP

LAN/WAN

Table and

Buffering

Packet

 

Lookup

 

Forwarding

 

LAN/WAN Interfaces

 

 

Media Cards

 

Figure 7. GRF 400

In conventional routers, each packet is processed at each gateway (also called hop) along a path. The processing is done at the Layer 3 level (see Figure 8 on page 14) and requires a router’s CPU to process both the packet and the route information. Conventional routers use shared resources, which leads to congestion and poor scalability and performance. Software-based route-table lookups can be very slow, if the route-table is not in cache.

Router Node 13

2

5

Router

Unswitched Data

Switched Data

1

 

4

 

Router

 

3

 

 

 

 

Router

Disadvantages:

Shared data paths

All processing done on Layer 3

Slow softwarebased processing

Layer 3

Layer 2

 

 

 

Router

Router

 

 

 

 

 

 

Process

 

 

Process

 

 

 

 

 

 

......Packets

 

 

......

 

 

 

 

 

 

 

 

Packets

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

.........

 

....

............

....

.....

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure 8. Conventional Routers

The SP Switch Router provides near wire-speed packet forwarding while using standard routing protocols. This ensures interoperability with other network technologies and does not require a specific network architecture, such as ATM. It works equally well in large and small networks. At each hop where a routing switch is used, routes are processed at Layer 3 but the packet is forwarded at Layer 2 (see Figure 9 on page 15). In the case of the SP Router, the route processing is done through hardware, so all processing is done at near-wire speed.

14 IBM 9077 SP Switch Router: Get Connected to the SP Switch

 

Unswitched Data

 

Switched Data

2

5

 

 

 

Router

Examples:

 

1

 

4

Ascend GRF, SP Switch Router

 

 

Router

Cisco 12000

 

 

3

 

 

 

 

 

 

 

 

 

 

Router

 

 

 

 

 

Disadvantages:

 

 

 

 

Hardware can be hard to upgrade *

 

 

 

 

Reduced routing functions *

 

Switch

 

Switch

 

Layer 3

Process Route

 

Process Route

Advantages:

 

 

 

 

Behave like traditional router

Layer 2 ..... ..... ......... ..... .....

Not dependent on a network architecture

Interoperability

Figure 9. Switched Routers

Other advantages of using GRF are as follows:

Availability of a redundant power supply

Availability of a redundant fan

Availability of a hot-swappable power supply

Availability of a hot-swappable fan (model 16S only)

Availability of hot-swappable media adapters (to connect to networks)

Scalability of up to 4 or 16 media adapters, depending on the GRF model

Perhaps the greatest advantage of using the GRF is improved price/performance. As previously mentioned, the GRF is a dedicated router, and as such it is much more cost effective for routing IP traffic than using dedicated RS/6000 SP node.

2.1.6 Overview of Supported Routing Protocols

In addition to static routes, various routing protocols are available on the GRF, as follows:

RIPRouting Information Protocol Version 1 or 2 (RIP 1 or 2)

OSPFOpen Shortest Path First

IS-ISIntermediate System to Intermediate System (an OSI gateway protocol)

Router Node 15

MulticastIP Multicast and OSPF Multicast

EGPExterior Gateway Protocol

BGPBorder Gateway Protocol Version 3 or 4 (BGP 3 or 4)

More details about the various protocols are in Section 2.2.2, “Supported Routing Protocols” on page 20.

2.1.7 Media Adapters At-a-Glance

Available IP forwarding media cards are:

1-port 100 Mbyte/s Switch Adapter

8-port 10/100 Mbit/s Ethernet cards

2-port 155 Mbit/s OC-3 ATM (Asynchronous Transfer Mode UNI 3.0/3.1)

1-port 622 Mbit/s OC-12 ATM

4-port 100 Mbit/s FDDI (Fiber Distributed Data Interface)

1-port 800 Mbit/s HIPPI (High Performance Parallel Interface)

2-port 52 Mbit/s or 45 Mbit/s DS-3 HSSI (High Speed Serial Interface)

1-port 155 Mbit/s IP/SONET OC-3c

More details are in Section 2.3.8, “Other Media Cards” on page 39.

2.1.8 Benefits of the GRF

The crosspoint switch is a nonblocking crossbar. This architecture is faster than an RS/6000 SP node, in which media adapters communicate through a shared microchannel bus.

To take advantage of the fast I/O provided by the crosspoint switch, fast route table access time is required. The GRF can store up to 150,000 routes in memory on each media card, while an RS/6000 SP node can store only hundreds. It is said that you need about 50,000 routes for the whole Internet. This means that the GRF is able to retrieve a route faster than an RS/6000 SP node.

The GRF is able to route up to 2.8 million packets per second for the 4-slot model and 10 million packets per second for the 16-slot model.

All the media adapters on the GRF are hot-pluggable. This differs from using an RS/6000 SP node as your router. Should any network adapter on the RS/6000 SP node fail, the node has to be brought down to replace the faulty

16 IBM 9077 SP Switch Router: Get Connected to the SP Switch

adapter. As a result, other network adapters are brought down as well. Bringing down the router will impact all the networks in the location.

Each RS/6000 SP is allowed to connect to multiple SP Switch Router Adapters, and it does not matter if these adapters are on different GRFs. Connecting multiple SP Switch Router adapters to either different partitions in an RS/6000 SP or to different RS/6000 SPs allows them to communicate with each other and with the other GRF media adapters via the SP Switch.

Attention

The SP Switch Router model 04S can support four media cards such as FDDI or ATM. The SP Switch Router model 16S can support 16. All card slots could be occupied by SP Switch Router adapters; this means a maximum of 4 SP Switch Router adapters for model 04S and a maximum of 16 SP Switch Router adapters for model 16S.

Note: The number of packets that the GRF can route per second depends on the following:

The type of media adapter

The size of the packet

2.1.9 Price Comparison

Figure 10 on page 18 shows a price comparison between an RS/6000 SP node solution and a GRF based solution for three sample configurations.

Router Node 17

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

9077-04S with one SP Switch Router Adapter

 

$ 53,000

 

 

 

 

 

 

 

HIPPI Adapter

 

 

 

13,500

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

9077-04S

 

 

 

 

 

 

 

 

 

 

 

 

 

with one SP Switch Router Adapter

 

 

66,500

$ 53,000

 

 

 

 

 

 

135 MHz

HIPPI A apter

 

 

 

48,000

 

 

 

 

 

 

 

 

 

19,000

 

 

 

 

 

 

Wide Node

9077-04S

 

 

 

3,200

 

 

 

 

 

 

64 MB memory

with one SP Switch Router Adapter

 

72,000

 

$ 53,000

 

 

 

 

4.5 GB Disk

 

1,950

 

 

 

 

 

HIPPI Adapter

 

 

 

 

 

 

 

 

 

48,000

 

 

 

 

 

Ethernet 135 MHz Wide Node

 

 

595

 

20,000

 

 

 

 

 

64 MB memory

 

 

10,000

3,200

 

 

 

 

 

 

 

 

 

 

 

 

SP Switch Adapter

 

 

 

 

17,500

1,950

 

73,000

 

 

 

 

 

4.5 GB Disk

 

 

 

 

 

 

 

 

 

HIPPI Adapter

135 MHz Wide Node

 

 

 

 

595

 

48,000

 

 

 

 

 

Ethernet

 

 

 

81,245

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

SP Switch

Adapter64 MB memory

 

 

 

10,000

 

3,200

 

 

 

 

 

4 FDDI 1-port SAS4.5GBAdaptersDisk *

 

 

 

 

15,980

 

1,950

 

 

 

 

 

 

 

 

 

 

 

 

 

595

 

 

 

 

 

 

Ethernet

 

 

 

 

75,730

 

 

 

 

 

 

 

 

 

 

 

 

10,000

 

 

 

 

 

 

SP Switch Adapter

 

 

 

 

 

 

 

 

 

 

 

 

2 ATM 155 Adapters *

 

 

 

 

 

5,390

 

 

 

 

 

 

 

 

 

 

 

 

 

69,135

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure 10. Price Comparison

These price comparisons are based on US prices as of March 1998. In other countries these prices may be different. The basic message of these charts is that the solutions based on the GRF could be quite competitive and will quite often be cheaper than the conventional configurations.

Let us look, for example for a solution connecting an RS/6000 SP via HIPPI to a mainframe system. The first chart shows that a GRF solution is cheaper than adding a dedicated node for the HIPPI connection to your system, apart from the fact that the GRF solution is the better choice from a performance point of view.

It is nearly the same if you need a connection to an FDDI network. One GRF FDDI media card offers four independent singlering connections. An offer based on an dedicated SP node is more expensive than a GRF solution.

Our example on the third chart focuses on an ATM connection. In this case, the RS/6000 SP node based solution is the more reasonable solution, if you consider only the price. But the difference is not that much; the growth path with the GRF-based offer will be better than the node solution.

2.2 GRF Software

The software functionality of the GRF is distributed between the Router Manager on the IP Switch Control Board (see also Section 2.3.2, “GRF Features” on page 26) and the individual media cards. While the Route

18 IBM 9077 SP Switch Router: Get Connected to the SP Switch

Manager updates the system routing tables and performs other administrative functions, the intelligent processors on each media card perform all routing functions. This design supports efficient distributed processing of router operations.

2.2.1 IP Protocol

The GRF supports IP datagram routing between major types of standard media. The implementation conforms with IP Version 4 and routing specifications described in Internet RFCs.

Each media card has a complete set of route and Address Resolution Protocol (ARP) information contained in the program memory space of the card’s on-board processor. IP packets are buffered in large transmit and receive buffers from which they are transmitted across the central switch fabric to the destination media. Any difference in MTU size (large MTU to smaller MTU) is handled by packet fragmentation as specified in the IP standard. Logic on the destination media is responsible for any media-specific processing of the packet, such as producing 53-byte cells for ATM.

Data Forwarding

Individual media cards maintain their own route tables, perform lookups, and autonomously handle the passing of datagrams to other media cards for export, without intervention of the Router Manager. Layer-3 decisions are local to each card.

Route Table Implementation

Critical to providing sustained performance in a highly dynamic environment are the cacheless route table and route lookup implementation. Each card carries a complete copy of the route table and can support up to 150,000 entries.

Keeping pace with significant advances in routing, the GRF also supports variable-length subnet masking and route aggregation.

Router Node 19

Subnet Masking/Supernetting

Variable length subnet masking is a classless addressing scheme for interdomain IP packet routing. It is a way to more efficiently manage the current 32-bit IP addressing method. Subnet masks let sites configure the size of their subnets based on the site needs, not on the arbitrary Class A, B, and C structure originally used in the Internet addressing. Class-based addressing restricts the boundary to the 8-bit boundaries and is implicitly derived from the first eight bits of the network address. The new addressing method allows the network portion of an IP address to be separated from the host portion of the address at any point within the 32-bit address space. This expanded boundary is called the "netmask" and is explicitly provided to the router along with the network address information. Class-based addressing restricts the boundary to the 8-bit boundaries and is implicitly derived from the first eight bits of the network address.

Subnet masking offers a number of benefits by extending the current address space. By eliminating implicit netmask assignments, addresses can now be assigned from any unused portion of the entire 32-bit address range rather than from within a specific subset of the space previously called a class. Since it hides multiple subnets under a single network address, this method is called supernetting.

Classless addressing allows the network administrator to further apportion an assigned address block into smaller network (or host) segments based on powers of two (2, 4, 8, 16 networks, for example). Knowledge of the apportioned segments need not be communicated to exterior peers. They need only a single pointer to the entire address block. Not only does subnet masking better utilize address space, but implemented properly it results in significantly smaller routing tables.

2.2.2 Supported Routing Protocols

In the days of a single Internet, core groups of independent networks were called autonomous systems. We will use the term autonomous systems (AS) in the following description of protocols. The routing protocols supported on the GRF can be divided into two classes: Interior routing protocols or interior gateway protocols (IGPs) and Exterior routing protocols (EGPs).

Interior routing protocols

Interior routing protocols are used to exchange routing information between routers within a single autonomous system. They are also used by routers that run exterior protocols to collect network reachability

20 IBM 9077 SP Switch Router: Get Connected to the SP Switch

information for the autonomous system. Here is the list of interior protocols supported by the GRF:

RIP

The Routing Information Protocol (RIP), as delivered with most UNIX systems, is run by the routing deamon routed. During the startup of routed a request for routing updates is issued. After that, the daemon listens for responses to the request. Systems that are configured to supply RIP information hear this request and respond with update packets based on the information in the system’s routing table. The update packets contain the destination addresses from the routing table and the routing metrics associated with each destination. Update packets are send on request and periodically to keep routing information accurate.

OSPF

Open Shortest Path First (OSPF) is defined by RFC 2178 (Request for Comments). It is a link-state protocol and very different from RIP. A router running RIP shares information about the entire network with its neighbors. A router running OSPF shares information about its neighbors with the entire network. The "entire network" means, at most, a single autonomous system. OSPF further defines a hierarchy of routing areas within an autonomous system:

Areas

These are sets of networks within a single autonomous system that have been grouped together. The topology of an area is hidden from the rest of the autonomous system, and each area has a separate topology database. Routing within the autonomous system takes place on two levels, depending on whether the source and destination of a packet reside in the same area (intra-area routing) or different areas (inter-area routing).

Intra-area routing is determined only by an area’s own topology. That is, the packet is routed solely on information obtained within the area.

Inter-area routing is always done via a backbone.

The dividing of an autonomous system into areas enables a significant reduction in the volume of routing traffic required to manage the routing database for a large autonomous system.

Backbone

The backbone consists of those networks not contained in any area, their attached routers, and routers that belong to multiple areas.

Router Node 21

Every area must connect to the backbone, because the backbone is responsible for the distribution of routing information between areas. The backbone itself has all the properties of an area. Its topology is separate from that of other areas.

Subarea

A subarea has only one area border router, which means that there is only one route out of the area. In this case, the area border router does not need to advertise external routes to the other routers within the subarea. It can simply advertise itself as the default route.

The sequence of operations performed by OSPF routers is as follows:

1.Discovering OSPF neighbors

2.Electing the Designated Router

3.Forming adjacencies

4.Synchronizing databases

5.Calculating the routing table

6.Advertising Link States

Routers go through these steps when they first come up, and repeat them in response to the events that occur in the network. Each router must perform each of these steps for each network it is connected to, except for the calculation of the routing table. Each router generates and maintains a single routing table for all networks.

Multicast

IP Multicast

The GRF supports IP multicast routing per RFC 1112 and some components of RFCs 1301 and 1469. The implementation includes the IP multicast kernel modifications, dynamic route support and mrouted (multicast route daemon).

Data that arrives at a GRF interface is duplicated and forwarded to multiple destination interfaces. The multicast packet’s destination address is a Class D address. A lookup of the multicast route table returns a list of Virtual Interfaces (VIFs) to which the packet is sent.

OSPF Multicast

The GRF uses the multicast capability of OSPF Version 2, as described in RFC 1583 and RIP Version 2, for communications between routers.

22 IBM 9077 SP Switch Router: Get Connected to the SP Switch

Host extensions for IP multicasting as described in RFC 1112 are also provided. The Router Manager acts as a host and uses the

Internet Group Management Protocol (IGMP), version 2, to add and delete its membership in multicast groups. Accordingly, the Route Manager "joins" the appropriate routing protocol host groups for OSPF and RIP.

IS-IS

Intermediate System to Intermediate System (an OSI gateway protocol) is a protocol similar to OSPF: it also uses a Link State, Shortest Path First algorithm. However, IS-IS is an OSI protocol used for routing Connectionless Network Protocol (CLNP) packets within a routing domain. CLNP is the OSI protocol most comparable to IP.

Exterior Routing Protocols

Exterior Routing Protocols are used to exchange routing information between routers in different autonomous systems. Here is the list of exterior routing protocols supported by the GRF:

EGP

The Exterior Gateway Protocol (EGP) is the protocol used for exchange of routing information between exterior gateways (not belonging to the same autonomous system).

EGP gateways may only forward reachability information for networks within their autonomous system. This routing information must be collected by the EGP gateway, usually via an GP).

BGP

Border Gateway Protocol (BGP) is the leading exterior routing protocol of the Internet and is replacing EGP as the exterior protocol of choice. It is based on the OSI InterDomain Routing Protocol (IDRP). BGP supports policy-based routing, which uses nontechnical reasons (for example, organizational, political, or security considerations) to make routing decisions. Thus, BGP enhances an autonomous system’s ability to choose between routes and to implement routing policies without relying on a central routing authority.

2.2.3 Filtering

IP filtering supports specific permit or deny decisions for each instance of a filter (per logical interface). The criteria within each filter may include any combination of the following:

• Protocol (ICMP, TCP, UDP)

Router Node 23

Source address

Destination address

Protocol port number (single number, or range, or ranges) for TCP and UDP

Established TCP connections

2.2.4 System Management

The GRF currently supports the Simple Network Management Protocol

(SNMP) Version 1, which provides a mechanism for remote query or setting operational parameters for the device. Media cards collect network statistics, which can be reported to network management packages via the Router Manager on the IP Switch Control Board. In addition to collection statistics and other management information, the SNMP agent is also capable of issuing traps. For more information, see Section 2.4.5, “SP Extension Node SNMP Manager” on page 58.

2.3 GRF Hardware

As already mentioned, the unique GRF switching architecture is specially designed for high-performance packet forwarding. The following sections give you more details about the various hardware components.

2.3.1 GRF Block Diagram

Figure 11 on page 25 shows the two models: the 4-slot and the 16-slot model.

24 IBM 9077 SP Switch Router: Get Connected to the SP Switch

 

 

 

 

 

 

 

 

 

 

19"

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Cooling Fan Drawer

 

 

 

 

 

 

21"

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

19"

Forwarding Media Card

Forwarding Media Card

Forwarding Media Card

Forwarding Media Card

Forwarding Media Card

Forwarding Media Card

Forwarding Media Card

Forwarding Media Card

Control Board IP Switch Board Forwarding Media Card

Forwarding Media Card

Forwarding Media Card

Forwarding Media Card

Forwarding Media Card

Forwarding Media Card

Forwarding Media Card

Forwarding Media Card

Power

IP Switch Control Board

Supply

IP Forwarding Media Card

IP

IP

IP

IP

IP

IP

IP

IP

IP

IP

IP

IP

IP

IP

IP

IP

 

IP Forwarding Media Card

Power

 

 

 

 

Power

 

 

 

Power

 

 

 

IP Forwarding Media Card

 

 

 

 

 

 

 

 

 

 

Supply

IP Forwarding Media Card

 

 

 

 

Supply

 

 

 

Supply

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

5.25"

GRF 400

 

 

 

 

 

 

 

 

GRF 1600

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure 11. GRF Models

The SP Switch Router model 9077 04S (GRF 400) can accommodate up to four media adapters.

The SP Switch Router model 9077 16S (GRF 1600) can accommodate up to 16 media adapters.

Each adapter enables the GRF to connect to one or more networks.

Each of the models has an additional slot for the IP Switch Control Board, which is used to control the router.

GRF 400

PartDescription

Cooling FansThese are located on the right side of the chassis and cannot be accessed without bringing down the GRF. The fans are redundant, allowing service to be deferred until it is convenient to bring down the GRF.

Media CardsThere are four media card slots on this chassis. They are slotted horizontally and are located at the bottom of the chassis.

IP Switch Control BoardThis board is located at the top of the four media slots and is also slotted horizontally.

Router Node 25

Power SupplyThe left side of the chassis is reserved for the two power supplies that are required for redundancy.

The failed power supply can be hot-swapped out of the GRF chassis.

GRF 1600

Part Description

Cooling FansThese are located at the top of the chassis, and can be accessed separately from the other parts of the GRF. The fan tray contains redundant fans and is hot-swappable.

Media CardsThere are 16 media card slots on this chassis. They are slotted vertically. Eight of the cards are on the left side of the chassis, and eight are on the right side.

IP Switch and

Control BoardThese boards are located in the middle of the 16 media slots and are also slotted vertically.

Power SupplyThe base of the chassis is reserved for the two power supplies that are required for redundancy. The failed power supply can be hot-swapped out of the GRF chassis.

2.3.2 GRF Features

GRF has the following features:

Redundant power supply

Should any power supply fail, a message is sent to the control board. The power supply will automatically reduce its output voltage if the temperature exceeds 90 °C (194 °F). If the voltage falls below 180V, the GRF automatically shuts down.

Hot-swappable power supply

The faulty power supply can be replaced while the GRF is in operation.

Redundant fan

For the GRF 1600 model, if one fan breaks down, a message is sent to the control board.

For both models, when the temperature reaches 53°C (128 °F), an audible alarm sounds continuously, and a message is sent to the console and logged into the message log.

26 IBM 9077 SP Switch Router: Get Connected to the SP Switch

If the temperature exceeds 57.5 °C (137 °F), the GRF does an automatic system shutdown.

Hot-swappable fan

For the GRF 16S model, the cooling fan can be replaced while the GRF is in operation.

Hot-swappable adapters

There are two types of adapters on the GRF: the media adapters and the IP Switch Control Board.

The media adapters are independent of each other and can be replaced or removed without affecting any other adapter or the operation of the GRF.

However, the IP Switch Control Board is critical to the GRF. Should this board be unavailable, the router fails.

Crosspoint switch

The crosspoint switch is a 16x16 (16Gb per second) or 4x4 (4Gb per second) crossbar switch for the GRF 16S and GRF 04S, respectively, see Figure 12 on page 27. It is the I/O path used when the media adapters need to communicate with each other.

Serial Interface

 

 

 

 

r

QBRT

RT

 

 

r

 

 

 

 

fe

 

 

fe

 

 

 

f

 

 

 

f

 

 

B

u

 

 

 

 

u

 

T

x

 

 

 

 

x

B

 

 

 

 

CPU

 

 

 

 

 

 

 

 

R

 

 

 

 

 

 

Media Interface

 

 

 

 

 

 

 

 

 

ATM OC-3c

MM

 

 

 

 

 

 

 

 

IP/SONET OC-3c

MM

 

 

 

CPU

............

..........

Combus

80 Mb/s bus

User Interface

Out-of-Band

 

Dynamic Routing Engine (Route Manager)

Configuration/Control

16MB Rx / 16MB Tx Buffers

Speed decoupling

WAN delays

QBRT Route Table Lookup

Times range from 1-2.5 µs

On-Board Processor

IP Header

Route decisions

4 Gb/s Switch

Figure 12. GRF Architecture

Some parts in Figure 12 need to be explained:

Router Node 27

1.Normally, all media cards have a 4 MB send buffer and a 4 MB receive buffer, except the SP Switch Adapter card, which has a 16 MB buffer size for each buffer. See also Section 2.3.5, “Characteristics of GRF Media Cards” on page 36 and Figure 19 on page 37.

2.Quick Branch Routing Technology (QBRT) is a hardware-assisted route table lookup. Route lookup times range from 1 - 2.5 µs with up to 150,000 next-hop routes in the table. Not all media cards use QBRT. Cards that do not use QBRT use a microcode lookup.

The benefit of this architecture is that the entire route table can be stored locally on the media card and searched quickly. In the traditional cached route table method, a small number of routes can be stored and searched locally. However, when a large number of routes is desired, or the kind of traffic one would see on the Internet backbone arises, caching is inadequate. Inevitably, cache misses occur, and route table lookups are performed at a limited, central, shared resource.

Performance is enhanced even further with parallel processing of table lookups occurring on each media card, which is another technique that helps assure linear scalability. The router manager on the controller board, which also contains the switch fabric, maintains the master route table and distributes updates simultaneously to all installed media cards, even as the cards continue their forwarding functions.

3.On-Board Processor

4.Router management takes place on the IP Switch Control Board (see Section 2.3.3.2, “IP Switch Control Board Components” on page 33), based on a 166 Mhz Pentium processor. It is responsible for system monitoring, configuration management and the user interface.

5.The GRF communications bus (Combus) is an "out-of-band" data path for configuration, control and monitoring of media cards. The Combus connects the IP Switch Control Board to the media cards independent of the switch connection to each card. It is not used for routed data between the cards. Route update packets received on any media card are also sent across the Combus to the Route Manager and, therefore, do not have to compete with normal IP traffic. The Combus is a serial bus with a transferrate of 80 Mb/s and is FIFO-buffered. The Arbitration Logic is on the IP Switch Control Board.

2.3.2.1 Data Packet Processing

With the knowledge about the local routing functions of the media cards, we now look at Figure 13 on page 29 to see how a data packet is transferred from one media card to another.

28 IBM 9077 SP Switch Router: Get Connected to the SP Switch

 

External

 

 

External

 

 

Interface

 

 

Interface

 

Buffer

DMA

Buffer

Buffer

DMA

Buffer

CPU

CPU

QBRT

QBRT

Tx

Rx

Tx

Rx

DMA

DMA

 

 

 

 

 

Internal

 

 

Internal

Switch

 

 

 

Control Board

 

Interface

 

 

Interface

 

 

 

 

 

 

Dynamic

 

 

 

 

 

Route

 

 

 

 

 

Manager

Figure 13. Data Packet Transfer

The routing can be divided into the following steps:

1.A data packet is received by the media card.

2.The packet is transferred to the receive buffer by the DMA engine.

3.The CPU examines the header and gives the destination address to the route lookup hardware.

4.The QBRT finds the next hop in the route table.

5.A special header is added to the packet by the CPU. This header contains information for the downstream card to process the packet, as well as the next hop found by the QBRT.

6.The packet is then transferred to the internal serial interface by the DMA engine.

7.A connection to the downstream card is set up through the switch.

8.The serial stream is converted back to parallel format by the downstream card.

9.The packet is transferred to the transmit buffer by the DMA engine.

Router Node 29

10.The header is examined by the CPU, which uses the information to build a new header that will deliver the data across the media interface.

11.The DMA engine transfers the packet to the media interface.

12.The packet is transferred across the media.

2.3.2.2 Routing Packet Processing

The processing of packets with routing information is a little bit different from the data packet processing procedure as you can see in Figure 14.

 

External

 

 

External

 

 

Interface

 

 

Interface

 

 

DMA

 

 

DMA

 

Buffer

CPU

Buffer

Buffer

CPU

Buffer

QBRT

QBRT

Rx

Tx

Rx

Tx

DMA

DMA

 

 

 

 

 

Internal

 

 

Internal

Switch

 

 

 

Control Board

 

Interface

 

 

Interface

 

 

 

 

 

 

Dynamic

 

 

 

 

 

Route

 

 

 

 

 

Manager

Figure 14. Routing Packet Processing

These are the steps for processing routing packets:

1.A routing packet is received by the media card.

2.The packet is transferred to the receive buffer by the DMA engine.

3.The CPU examines the header and gives the destination address to the route lookup hardware.

4.The QBRT finds the next hop in the route table.

5.A special header is added to the packet by the CPU. This header contains information for the downstream card. The result of the hardware lookup determines whether the packet should be forwarded to the Router Manager.

30 IBM 9077 SP Switch Router: Get Connected to the SP Switch

6.The packet is then transferred to the Combus interface by the DMA engine.

7.The packet is sent to the IP Switch Control Board’s Router Manager across the Combus.

8.The Route Manager receives the packet and passes it to the dynamic routing software.

9.The packet is processed and global routing information is determined. 10.Route updates are broadcast across the Combus to all media cards

simultaneously.

11.Each card receives the update packet and makes changes to its route tables.

12.The packet is transferred across the media.

To ensure that dynamic routing packets are not dropped during times of heavy congestion, precedence features are used. Routing packets are given a high-priority tag and a user-configurable threshold for Tx buffers is maintained for high-priority traffic.

2.3.3 IP Switch and Control Board

The control board, also known as the IP Switch Control Board, is accessed through Telnet or a locally attached VT100 terminal. The IP Switch Control Board is supplied with the GRF and is necessary for its operation. The VT100 terminal is not supplied with the GRF. It is only needed for the installation of the GRF.

Using terminal emulation software instead of looking for a real VT 100 terminal may be an alternative. You can use your Control Workstation or one of your SP nodes. Install the ATE package (advanced terminal emulation) on your RS/6000 and establish a serial connection between the system and the router.

After installation, all future access to the GRF can be through Telnet to the IP Switch Control Board’s administrative Ethernet.

The IP Switch Control Board is identified as slot 66 in both GRF models. A sideview of the GRF 400 slot numbering scheme is shown in Figure 15 on page 32.

Router Node 31

Backplane

66

3

2

1

0

IP Switch Control Board

Slot numbers in decimal for media cards

Figure 15. Side View of GRF 400 Chassis with Slots Numbered

The GRF 1600 has 16 media slots. The control board is located in slot 66, as shown in Figure 16.

Backplane

 

 

 

Media cards

 

 

 

Slot numbers

0 1 2 3 4 5 6 7

66

8 9 10 11 12 13 14 15

 

Control board

 

Switch board

Figure 16. Top View of the GRF 1600 Chassis

The CPU in the IP Switch Control Board is a 166MHz Pentium processor and runs a variant of BSD UNIX as its operating system. For this reason, the GRF administrator is assumed to be proficient in UNIX.

The IP Switch Control Board is used to install, boot, and configure the router and its media adapters.

It is also used for the logging of messages, the dumping of memory and status, and to perform diagnostic checking of both the GRF and the media adapters.

32 IBM 9077 SP Switch Router: Get Connected to the SP Switch

+ 292 hidden pages