Size:
3 Mb
Download

IBM 9077 SP Switch Router: Get Connected to the SP Switch

Hajo Kitzhöfer, Steffen Eisenblätter, Uwe Untermarzoner

International Technical Support Organization

http://www.redbooks.ibm.com

SG24-5157-00

SG24-5157-00

International Technical Support Organization

IBM 9077 SP Switch Router:

Get Connected to the SP Switch

November 1998

Take Note!

Before using this information and the product it supports, be sure to read the general information in Appendix D, “Special Notices” on page 305.

First Edition (November 1998)

This edition applies to PSSP Version 2, Release 4 for use with AIX 4.3.1 and Ascend Embedded/OS Version 1.4.6.4.

Comments may be addressed to:

IBM Corporation, International Technical Support Organization Dept. HYJ Mail Station P099

522 South Road

Poughkeepsie, New York 12601-5400

When you send information to IBM, you grant IBM a non-exclusive right to use or distribute the information in any way it believes appropriate without incurring any obligation to you.

© Copyright International Business Machines Corporation 1998. All rights reserved

Note to U.S Government Users – Documentation related to restricted rights – Use, duplication or disclosure is subject to restrictions set forth in GSA ADP Schedule Contract with IBM Corp.

Contents

Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .ix

Tables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xi

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii The Team That Wrote This Redbook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii Comments Welcome . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv

Part 1. Introducing and Installing the GRF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Chapter 1. Dependent Node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.1 Dependent Node Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.2 Limitations of the Dependent Node. . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Chapter 2. Router Node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.1.1 Motivation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.1.2 Design Objectives. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.1.3 What is a Router. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.1.4 Routing without the GRF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.1.5 Routing with the GRF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.1.6 Overview of Supported Routing Protocols. . . . . . . . . . . . . . . . . . 15 2.1.7 Media Adapters At-a-Glance. . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.1.8 Benefits of the GRF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.1.9 Price Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.2 GRF Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.2.1 IP Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.2.2 Supported Routing Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.2.3 Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.2.4 System Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.3 GRF Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.3.1 GRF Block Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.3.2 GRF Features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.3.3 IP Switch and Control Board. . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 2.3.4 Memory Guidelines for the IP Switch Control Board . . . . . . . . . . 35 2.3.5 Characteristics of GRF Media Cards. . . . . . . . . . . . . . . . . . . . . . 36 2.3.6 SP Switch Router Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 2.3.7 Media Card Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 2.3.8 Other Media Cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 2.3.9 GRF Operating Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 2.4 PSSP Enhancements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

© Copyright IBM Corp. 1998

iii

2.4.1 SDR Enhancements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 2.4.2 New Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 2.4.3 Enhanced Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 2.4.4 Hardware Perspectives. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 2.4.5 SP Extension Node SNMP Manager. . . . . . . . . . . . . . . . . . . . . . 58 2.4.6 Dependent Node MIB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 2.4.7 Coexistence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 2.4.8 Partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

2.5 Planning for the GRF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 2.6 Planning for the Dependent Node. . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 2.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

Part 2. Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

Chapter 3. Installation and Configuration . . . . . . . . . . . . . . . . . . . . . . . 69 3.1 Initial Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 3.2 Pre-Installation Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 3.2.1 Order of Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 3.3 Installing an SP Switch Router Adapter Card . . . . . . . . . . . . . . . . . . . 75 3.3.1 Installation Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 3.3.2 Installing the PCMCIA Spinning Disk . . . . . . . . . . . . . . . . . . . . . 76 3.4 Attaching SP Switch Router Cables . . . . . . . . . . . . . . . . . . . . . . . . . . 79 3.4.1 Ethernet Cable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 3.4.2 SP Switch Cable. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 3.4.3 Procedure for Connecting Cards to the SP Switch . . . . . . . . . . . 80 3.5 Configuration Required on the SP System . . . . . . . . . . . . . . . . . . . . . 81 3.5.1 Determining the Switch Connection for a Dependent Node. . . . . 82 3.5.2 Procedure to Get the Jack Number. . . . . . . . . . . . . . . . . . . . . . . 84 3.6 Multiple Frames for Multiple System Connections . . . . . . . . . . . . . . . 85 3.7 Step-by-Step Media Card Configuration . . . . . . . . . . . . . . . . . . . . . . . 86 3.7.1 Configuration Files and Their Uses . . . . . . . . . . . . . . . . . . . . . . . 86 3.8 Step 1. Check SNMP in the SP Switch Router System . . . . . . . . . . . . 88 3.9 Put SNMP Changes into Effect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 3.10 Step 2. Assign IP Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 3.10.1 Method 1: Use SP SNMP Manager - Recommended . . . . . . . . 89 3.10.2 Method 2: Edit /etc/grifconfig.conf - Optional . . . . . . . . . . . . . . 93 3.10.3 Putting grifconfig.conf Additions into Effect . . . . . . . . . . . . . . . . 95 3.11 Step 3. Change Profile Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 3.12 Step 4. Run dev1config . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 3.13 Step 5. Reset SP Switch Router System to Install Files . . . . . . . . . . 96 3.13.1 Saving Configuration Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 3.14 Verify an SP Switch Router Adapter Card on the Router . . . . . . . . . 97 3.14.1 Verify Media Card Operation Using ping . . . . . . . . . . . . . . . . . . 97

iv IBM 9077 SP Switch Router: Get Connected to the SP Switch

3.14.2 Check Media Card Status Using grcard . . . . . . . . . . . . . . . . . . 98 3.14.3 Reset Media Card Using grreset . . . . . . . . . . . . . . . . . . . . . . . . 99 3.14.4 Using grstat to Display GRF Statistics . . . . . . . . . . . . . . . . . . . 99 3.15 Bringing the SP Switch Router Adapter Card Online with the SP . . 100 3.15.1 Checking Connectivity to the SP System . . . . . . . . . . . . . . . . 101

Chapter 4. Configuration of IP-Forwarding Media Cards . . . . . . . . . . 105 4.1 Ethernet 10/100Base-T Configuration. . . . . . . . . . . . . . . . . . . . . . . . 105 4.1.1 Physical and Logical Interfaces . . . . . . . . . . . . . . . . . . . . . . . . 105 4.1.2 Configuration File and Profile Overview . . . . . . . . . . . . . . . . . . 106 4.1.3 Installing Configurations or Changes . . . . . . . . . . . . . . . . . . . . 107 4.1.4 Assign IP Addresses - grifconfig.conf . . . . . . . . . . . . . . . . . . . . 107 4.1.5 Specify Ethernet Card Parameters . . . . . . . . . . . . . . . . . . . . . . 108 4.1.6 Some maint Commands for the Ethernet Media Cards . . . . . . . 109 4.2 ATM OC-3c Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 4.2.1 Physical and Logical ATM Interfaces . . . . . . . . . . . . . . . . . . . . 110 4.2.2 Installing Configurations or Changes . . . . . . . . . . . . . . . . . . . . 113 4.2.3 Configuration Files and Profiles . . . . . . . . . . . . . . . . . . . . . . . . 113 4.2.4 Assign IP Addresses - grifconfig.conf . . . . . . . . . . . . . . . . . . . . 114 4.2.5 Specify ATM Card Parameters . . . . . . . . . . . . . . . . . . . . . . . . . 115 4.2.6 Configuring PVCs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 4.2.7 Some maint Commands for the ATM OC-3c Media Card . . . . . 116 4.2.8 Using grrt to Display the Route Table . . . . . . . . . . . . . . . . . . . . 118 4.2.9 Using grstat to Display GRF Statistics . . . . . . . . . . . . . . . . . . . 119 4.3 ATM OC-12c Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 4.3.1 Physical and Logical ATM Interfaces . . . . . . . . . . . . . . . . . . . . 119 4.3.2 Installing Configurations or Changes . . . . . . . . . . . . . . . . . . . . 120 4.3.3 Configuration Files and Profiles . . . . . . . . . . . . . . . . . . . . . . . . 120 4.4 FDDI Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 4.4.1 Separate Networks versus Bridging . . . . . . . . . . . . . . . . . . . . . 126 4.4.2 Naming the FDDI Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 4.4.3 Physical Interface Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 4.4.4 GRF Interface Name. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 4.4.5 Configuration Files and Profiles . . . . . . . . . . . . . . . . . . . . . . . . 128 4.4.6 Assign IP Addresses - grifconfig.conf . . . . . . . . . . . . . . . . . . . . 129 4.4.7 Specify FDDI Card Parameters. . . . . . . . . . . . . . . . . . . . . . . . . 130 4.4.8 Installing Configurations or Changes . . . . . . . . . . . . . . . . . . . . 130 4.4.9 Some maint Commands for the FDDI Media Card . . . . . . . . . . 131 4.4.10 Using grrt to Display the Route Table . . . . . . . . . . . . . . . . . . . 132 4.4.11 Using grstat to Display GRF Statistics . . . . . . . . . . . . . . . . . . 133 4.5 HIPPI Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 4.5.1 Introduction to HIPPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 4.5.2 HIPPI Configuration Options . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

v

4.5.3 Physical and Logical Interfaces . . . . . . . . . . . . . . . . . . . . . . . . 139 4.5.4 Configuration Files and Profiles . . . . . . . . . . . . . . . . . . . . . . . . 140 4.5.5 Installing Configurations or Changes . . . . . . . . . . . . . . . . . . . . 141 4.5.6 Some maint Commands for the HIPPI Media Card . . . . . . . . . . 141 4.6 Configuring Bridging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 4.6.1 GRF Bridging Implementation. . . . . . . . . . . . . . . . . . . . . . . . . . 142 4.6.2 Simultaneous Routing and Bridging . . . . . . . . . . . . . . . . . . . . . 143 4.6.3 Configuration Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 4.6.4 Interoperability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 4.6.5 Spanning Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 4.6.6 Bridge Filtering Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 4.6.7 Fragmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 4.6.8 Spamming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 4.6.9 Bridging Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 4.6.10 Management Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 4.6.11 Configuration File and Profile Overview . . . . . . . . . . . . . . . . . 148 4.6.12 Bridging ATM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 4.6.13 Bridging FDDI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 4.6.14 Bridging Ethernet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155

Chapter 5. Single RS/6000 SP and Single SP Switch Router . . . . . . . 157 5.1 Single SP Partition and Single SP Switch Router Adapter Card . . . . 157 5.1.1 SP Switch - Ethernet Connection . . . . . . . . . . . . . . . . . . . . . . . 157 5.1.2 SP Switch - FDDI Connection . . . . . . . . . . . . . . . . . . . . . . . . . . 162 5.1.3 SP Switch - ATM Connection . . . . . . . . . . . . . . . . . . . . . . . . . . 167 5.1.4 SP Switch - FDDI Connection (Distinct FDDI Networks) . . . . . . 174 5.1.5 SP Switch - FDDI Connection in an ADSM Environment. . . . . . 185 5.2 Single SP Partition and Multiple SP Switch Router Adapter Cards . . 187 5.2.1 Configuration of a Dual SP Switch Router Connection . . . . . . . 187 5.2.2 Complex Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 5.2.3 Recovery Procedure for an SP Switch Adapter Card Failure. . . 196 5.3 Multiple SP Partition and Multiple SP Switch Router Adapter Cards . 197

Chapter 6. Multiple RS/6000 SPs and One SP Switch Router . . . . . . 203 6.1 RS/6000 SP Switch - RS/6000 SP Switch Connection . . . . . . . . . . . 203 6.2 Sharing Network Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207

Chapter 7. Multiple RS/6000 SPs and Multiple GRFs . . . . . . . . . . . . . 209 7.1 ATM OC-3c Backbone Connection . . . . . . . . . . . . . . . . . . . . . . . . . . 209 7.1.1 ATM OC-3c Backbone - Using One Port . . . . . . . . . . . . . . . . . . 210 7.1.2 ATM OC-3c Backbone - Using Two Ports . . . . . . . . . . . . . . . . . 215 7.2 ATM OC-12c Backbone - One Port. . . . . . . . . . . . . . . . . . . . . . . . . . 222 7.3 HIPPI Backbone Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227

vi IBM 9077 SP Switch Router: Get Connected to the SP Switch

Appendix A. Laboratory Hardware and Software Configuration . . . . 233 A.1 Node and Control Workstation Configuration . . . . . . . . . . . . . . . . . . . . . 233 A.1.1 Hard Disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235 A.1.2 Software Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239 A.1.3 Network Options and Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253 A.2 SP Switch Pool Size Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255 A.3 7025-F50 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256 A.4 SP IP Switch Router Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258

Appendix B. GRF Configuration Files . . . . . . . . . . . . . . . . . . . . . . . . . . 261 B.1 /root/.profile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261 B.2 /etc/Release . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263 B.3 /etc/bridged.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264 B.4 /etc/fstab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267 B.5 /etc/grarp.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267 B.6 /etc/gratm.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268 B.7 /etc/grclean.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274 B.8 /etc/grclean.logs.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275 B.9 /etc/grdev1.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277 B.10 /etc/grifconfig.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282 B.11 /etc/grlamap.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284 B.12 /etc/grroute.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285 B.13 /etc/hosts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 B.14 /etc/inetd.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 B.15 /etc/motd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287 B.16 /etc/rc.local . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287 B.17 /etc/snmpd.conf. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288 B.18 /etc/syslog.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291 B.19 /etc/ttys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292

Appendix C. Hardware and Software Information . . . . . . . . . . . . . . . . 295 C.1 The Front Panel of the SP Switch Router Adapter Card - Operational. . 295 C.2 SP Switch Router Adapter Media Card LEDs. . . . . . . . . . . . . . . . . . . . . 296 C.3 SP Switch Router Adapter Media Card - Bootup . . . . . . . . . . . . . . . . . . 297 C.4 Connectors and Receptacles for Different Media . . . . . . . . . . . . . . . . . . 298 C.5 Chip Interconnection on the TBS Board . . . . . . . . . . . . . . . . . . . . . . . . . 298 C.6 Updating Router Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299

C.6.1 The SP Switch Router as an IBM Product . . . . . . . . . . . . . . . . . . . 299 C.6.2 Obtaining New Machine Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300 C.6.3 Support for Code Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300 C.6.4 Sample Steps to Upgrade the System Software . . . . . . . . . . . . . . 300 C.6.5 Sample Execution of grf_update Script . . . . . . . . . . . . . . . . . . . . . 301

vii

Appendix D. Special Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305

Appendix E. Related Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309 E.1 International Technical Support Organization Publications . . . . . . . . . . 309 E.2 Redbooks on CD-ROMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309 E.3 Other Publications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309

How to Get ITSO Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311 How IBM Employees Can Get ITSO Redbooks . . . . . . . . . . . . . . . . . . . . . . . 311 How Customers Can Get ITSO Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . 312 IBM Redbook Order Form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313

List of Abbreviations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317

ITSO Redbook Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323

viii IBM 9077 SP Switch Router: Get Connected to the SP Switch

Figures

1. SP Switch Router. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2. Functional Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 3. Typical Router Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 4. Table-Based Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 5. Routing without GRF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 6. Routing with GRF. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 7. GRF 400 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 8. Conventional Routers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 9. Switched Routers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 10. Price Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 11. GRF Models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 12. GRF Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 13. Data Packet Transfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 14. Routing Packet Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 15. Side View of GRF 400 Chassis with Slots Numbered . . . . . . . . . . . . . . . . 32 16. Top View of the GRF 1600 Chassis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 17. IP Switch Control Board. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 18. System RAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 19. SP Switch Router Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 20. Hardware Perspectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 21. Action Menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 22. Hardware Notebook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 23. System Partition Aid Perspectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 24. System Partition Aid Notebook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 25. Coexistence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 26. Partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 27. The Laboratory Hardware Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 28. Connecting the GRF to the SP Switch and the CWS . . . . . . . . . . . . . . . . 69 29. Connecting the GRF to the Frame. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 30. Connecting the GRF Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 31. SP System Administrative Ethernet Connections . . . . . . . . . . . . . . . . . . . 80 32. Switch Port Assignments in Supported Frame Configurations . . . . . . . . . 83 33. Node Numbering for an SP System. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 34. How Frames Enable Connections to Multiple SP Switches. . . . . . . . . . . . 86 35. Components in the SP Switch Router Adapter Card’s Interface Name . . . 93 36. Components of the Ethernet Interface Name . . . . . . . . . . . . . . . . . . . . . 106 37. ATM OC-3c Physical and Logical Interfaces . . . . . . . . . . . . . . . . . . . . . . 110 38. Components in the ATM OC-3c Interface Name . . . . . . . . . . . . . . . . . . . 111 39. Components Forming a Virtual Path . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 40. ATM OC-12c Physical and Logical Interfaces . . . . . . . . . . . . . . . . . . . . . 120

© Copyright IBM Corp. 1998

ix

41. Master/Slave Connectors for SAS Interfaces . . . . . . . . . . . . . . . . . . . . . 122 42. A/B Connectors for DAS Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 43. Allowed SAS and DAS Configurations. . . . . . . . . . . . . . . . . . . . . . . . . . . 123 44. Optical Bypass Switch Attachments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 45. Dual Homing Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 46. Assigning Numbers to FDDI Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . 127 47. Physical Interface Numbering on the FDDI Media Card . . . . . . . . . . . . . 128 48. GRF Interface Name for FDDI Interfaces . . . . . . . . . . . . . . . . . . . . . . . . 128 49. HIPPI I-Field Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 50. Components in the HIPPI Interface Name. . . . . . . . . . . . . . . . . . . . . . . . 139 51. Interface Name for FDDI, Ethernet and ATM OC-3c Interfaces . . . . . . . 150 52. One Card - One SP Partition Sample Configuration . . . . . . . . . . . . . . . . 157 53. SP Switch - Ethernet Connection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 54. SP Switch - FDDI Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 55. SP Switch - ATM Connection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 56. SP Switch - FDDI Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 57. SP Switch - FDDI Connection (Bridging) . . . . . . . . . . . . . . . . . . . . . . . . . 180 58. SP Switch Router in an ADSM Environment . . . . . . . . . . . . . . . . . . . . . . 185 59. Connecting One SP Switch with Two SP Switch Router Adapter Cards . 187 60. Configuration with Dual SP Switch Router - SP Switch Connection . . . . 190 61. IP Traffic Flow When Issuing ping 192.168.13.1 on Node 6 . . . . . . . . . . 195 62. IP Traffic Flow When Issuing ping 192.168.13.1 on Node 10 . . . . . . . . . 195 63. IP Traffic Flow When Issuing ping 192.168.13.1 on Node 8 . . . . . . . . . . 196 64. Partition-to-Partition Connection with an SP Switch Router . . . . . . . . . . 198 65. Two RS/6000 SPs Connected to GRF 1600 . . . . . . . . . . . . . . . . . . . . . . 203 66. Sharing Network Resources between Two SPs . . . . . . . . . . . . . . . . . . . 207 67. Connection of Two SPs with Two SP Switch Routers . . . . . . . . . . . . . . . 209 68. SP Switch - ATM - SP Switch Connection . . . . . . . . . . . . . . . . . . . . . . . . 211 69. SP Switch - ATM Bridged - SP Switch Connection . . . . . . . . . . . . . . . . . 215 70. SP Switch - ATM OC-12c - SP Switch Connection . . . . . . . . . . . . . . . . . 223 71. SP Switch - HIPPI - SP Switch Connection . . . . . . . . . . . . . . . . . . . . . . . 228 72. Front Panel of the SP Switch Router Adapter Card with LEDs . . . . . . . . 295 73. The SP Switch Board. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299

x IBM 9077 SP Switch Router: Get Connected to the SP Switch

Tables

1. Memory Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 2. DependentNode Attributes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 3. DependentAdapter Attributes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 4. Additional SDR Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 5. New Commands (root Executable) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 6. New Commands (User Executable). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 7. endefnode Command Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 8. enrmnode Command Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 9. endefadapter Command Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 10. enadmin Command Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 11. splstnode Command Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 12. splstadapter Command Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 13. Enhanced Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 14. Configuration of SP Switch - Ethernet Connection . . . . . . . . . . . . . . . . . 159 15. Configuration of an SP Switch - FDDI Connection . . . . . . . . . . . . . . . . . 163 16. Configuration of SP Switch - ATM Connection . . . . . . . . . . . . . . . . . . . . 168 17. Configuration of SP Switch - FDDI Connection . . . . . . . . . . . . . . . . . . . . 175 18. Configuration of SP Switch - FDDI Connection (Bridging). . . . . . . . . . . . 181 19. Configuration of a Dual SP Switch Router Connection . . . . . . . . . . . . . . 187 20. Configuration of a Dual SP Switch Router - SP Switch Connection . . . . 191 21. Configuration of a Partition - Partition Connection . . . . . . . . . . . . . . . . . 199 22. Configuration of SP Switch - SP Switch Connection . . . . . . . . . . . . . . . . 204 23. Configuration of SP Switch - ATM - SP Switch . . . . . . . . . . . . . . . . . . . . 212 24. Configuration of SP Switch - ATM Bridged - SP Switch . . . . . . . . . . . . . 216 25. Configuration of SP Switch - ATM OC-12c - SP Switch . . . . . . . . . . . . . 224 26. Configuration of SP Switch - HIPPI - SP Switch . . . . . . . . . . . . . . . . . . . 228 27. Configuration of SP 21. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234 28. Configuration of SP 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235 29. Hard Disk Equipment of SP 21 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236 30. Hard Disk Equipment of SP 2 Part 1 of 2. . . . . . . . . . . . . . . . . . . . . . . . . 237 31. Hard Disk Equipment of SP 2 Part 2 of 2. . . . . . . . . . . . . . . . . . . . . . . . . 238 32. Software Levels on CWS and All Nodes Part 1 of 14 . . . . . . . . . . . . . . . 239 33. Software Levels on CWS and All Nodes Part 2 of 14 . . . . . . . . . . . . . . . 240 34. Software Levels on CWS and All Nodes Part 3 of 14 . . . . . . . . . . . . . . . 241 35. Software Levels on CWS and All Nodes Part 4 of 14 . . . . . . . . . . . . . . . 242 36. Software Levels on CWS and All Nodes Part 5 of 14 . . . . . . . . . . . . . . . 243 37. Software Levels on CWS and All Nodes Part 6 of 14 . . . . . . . . . . . . . . . 244 38. Software Levels on CWS and All Nodes Part 7 of 14 . . . . . . . . . . . . . . . 245 39. Software Levels on CWS and All Nodes Part 8 of 14 . . . . . . . . . . . . . . . 246 40. Software Levels on CWS and All Nodes Part 9 of 14 . . . . . . . . . . . . . . . 247

© Copyright IBM Corp. 1998

xi

41. Software Levels on CWS and All Nodes Part 10 of14 . . . . . . . . . . . . . . . 248 42. Software Levels on CWS and All Nodes Part 11 of 14 . . . . . . . . . . . . . . 249 43. Software Levels on CWS and All Nodes Part 12 of 14 . . . . . . . . . . . . . . 250 44. Software Levels on CWS and All Nodes Part 13 of 14 . . . . . . . . . . . . . . 251 45. Software Levels on CWS and All Nodes Part 14 of 14 . . . . . . . . . . . . . . 252 46. Network Options of CWS and All Nodes Part 1 of 3 . . . . . . . . . . . . . . . . 253 47. Network Options of CWS and All Nodes Part 2 of 3 . . . . . . . . . . . . . . . . 254 48. Network Options of CWS and All Nodes Part 3 of 3 . . . . . . . . . . . . . . . . 255 49. Network Options of 7025-F50 Part 1 of 3 . . . . . . . . . . . . . . . . . . . . . . . . 256 50. Network Options of 7025-F50 Part 2 of 3 . . . . . . . . . . . . . . . . . . . . . . . . 257 51. Network Options of 7025-F50 Part 3 of 3 . . . . . . . . . . . . . . . . . . . . . . . . 258 52. SP Switch Router Adapter Media Card LEDs . . . . . . . . . . . . . . . . . . . . . 296 53. SP Switch Router Adapter Media Card LEDs - RX/TX . . . . . . . . . . . . . . 296 54. SP Switch Router Adapter Media Card LEDs During Bootup . . . . . . . . . 297 55. Media Card Cables and Connectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298

xii IBM 9077 SP Switch Router: Get Connected to the SP Switch

Preface

The GRF is a high-performance switched IP Router which provides high-speed data communication links between IBM RS/6000 SP and external networks or hosts. It acts as a special-purpose SP node that routes IP traffic between SP nodes on the SP Switch and the outside world. Connected directly to the SP Switch, the router offers significantly improved SP I/O performance. When packaged with an IBM SP system, the GRF router is referred to as the SP Switch Router.

This redbook helps you install, tailor and configure the SP Switch Router, IBM machine type 9077. The SP Switch Router is also known as the "Gigarouter" or High Performance Gateway Node (HPGN).

The first part of the book gives an overview of the GRF architecture and how the router was integrated into the SP. It emphasizes the advantages of choosing a dedicated router node in some configurations, as opposed to using standard nodes for the routing task. This part also describes some routing fundamentals, particularly focusing on concepts like IPand switch-routing.

The second part presents sample configurations that were carefully chosen to match frequently occurring customer situations. The basic configurations shown are building blocks for more complex networking topologies that include the SP Switch Router and may inspire more sophisticated configurations. All configurations described were tested and provide some comparable performance figures.

This publication is intended to give IBM customers, system engineers, and marketing personnel a broad understanding of this new architecture and what it is used for.

The Team That Wrote This Redbook

This redbook was produced by a team of specialists from around the world working at the International Technical Support Organization, Poughkeepsie Center.

Dr Hajo Kitzhöfer is an Advisory International Technical Support Organization (ITSO) Specialist for RS/6000 SP at the Poughkeepsie Center. He holds a Ph.D. degree in electrical engineering from the Ruhr-University of Bochum (RUB). Before joining ITSO, he worked as an SP Specialist at the RS/6000 and AIX Competence Center, IBM Germany. He has worked at IBM

© Copyright IBM Corp. 1998

xiii

for eight years. His areas of expertise include RS/6000 SP, SMP, and Benchmarks. He now specializes in SP System Management, SP Performance Tuning and SP hardware.

Dr Steffen Eisenblätter is an AIX Software Specialist in the RS/6000 SP Software Support Center, Germany. He holds a Ph.D. degree in physics from the University of Leipzig. He joined IBM in 1997 and has focused on RS/6000 SP products and TCP/IP.

Uwe Untermarzoner is an RS/6000 SP Technical Support Specialist with IBM Germany. He joined IBM 1989. He has ten years of experience in AIX and five years of experience with the SP, mostly in the commercial environment. He joined IBM at 1989. His areas of expertise include AIX, RS/6000 SP, SMP, PSSP, Networking, Performance Tuning and Systems Management.

Thanks to the following people for their invaluable contributions to this project:

Ronald Linton

IBM PPS Lab Poughkeepsie

Gene Novitsky

Ascend Communications, Inc.

Frank May

IBM Worldwide RS/6000 SP Product Marketing

Wes Kinard

IBM RS/6000 Networking Technologies

Marcelo R. Barrios

International Technical Support Organization, Poughkeepsie Center

xiv IBM 9077 SP Switch Router: Get Connected to the SP Switch

Comments Welcome

Your comments are important to us!

We want our redbooks to be as helpful as possible. Please send us your comments about this or other redbooks in one of the following ways:

Fax the evaluation form found in “ITSO Redbook Evaluation” on page 323 to the fax number shown on the form.

Use the electronic evaluation form found on the Redbooks Web sites:

For Internet usershttp://www.redbooks.ibm.com For IBM Intranet usershttp://w3.itso.ibm.com

Send us a note at the following address:

redbook@us.ibm.com

xv

xvi IBM 9077 SP Switch Router: Get Connected to the SP Switch

Part 1. Introducing and Installing the GRF

© Copyright IBM Corp. 1998

1

2 IBM 9077 SP Switch Router: Get Connected to the SP Switch

Chapter 1. Dependent Node

This chapter provides an overview of a dependent node in RS/6000 SP. We start by defining the dependent node and the rationale behind its design.

1.1 Dependent Node Architecture

The Dependent Node Architecture refers to a processor or node, possibly not provided by IBM, for use with the RS/6000 SP.

Since a dependent node may not be a regular RS/6000 SP node, not all the functions of a node can be performed on it, which is why it is called "dependent". For example, it does not allow all the functions of the fault service (Worm) daemon, as other RS/6000 SP nodes with access to the SP Switch do.

The objective of this architecture is to allow the other processors or hardware to easily work together with the RS/6000 SP, extending the scope and capabilities of the system.

The dependent node connects to the RS/6000 SP Switch (but not to the earlier High Performance Switch, HiPS).

The SP Switch Router Adapter is the first product to exploit the Dependent Node Architecture.

1.2 Limitations of the Dependent Node

The following are limitations associated with use of the dependent node:

To use the dependent node in an RS/6000 SP requires the SP Extension Node SNMP Manager to be installed in the Control Workstation. The SP Extension Node SNMP Manager requires UDP port 162 in the Control Workstation. Other SNMP managers, such as Netview, also require this port. To allow the two SNMP managers to coexist, the SP Extension Node SNMP Manager must use an alternative UDP port.Dependent nodes are not allowed in Node Groups.

Only the 8-port and 16-port SP Switch are supported. The 8-port and 16-port High Performance Switch (the old SP Switch) are not supported.

The spmon command on the RS/6000 SP is not enhanced to support dependent nodes. Dependent nodes can only be viewed with the perspectives command.

© Copyright IBM Corp. 1998

3

The fault service daemon runs on all switch nodes in the RS/6000 SP, but not on the dependent node. Therefore, the dependent node does not have the full functionality of a normal RS/6000 SP Switch node.

The dependent node requires the SP Switch’s primary node to compute its switch routes. Therefore, the primary node must have at least PSSP 2.3 installed, otherwise the dependent node cannot work with the RS/6000 SP.

In the RS/6000 SP, SP Switch nodes occasionally send service packets from one node to the next to keep track of status and links. Sometimes these packets are sent indirectly through another switch node. As the dependent node is not a standard RS/6000 SP Switch node, it cannot be used to forward service packets to other nodes.

4 IBM 9077 SP Switch Router: Get Connected to the SP Switch

Chapter 2. Router Node

The first dependent node is actually a new SP Switch Router Adapter in a router. This chapter offers more details about the implementation.

Section 2.1, “Overview” on page 5 gives you an overview of SP Switch Router. This is probably the best to get an impression what the GRF is good for. Also a functionaland a price-comparison between using an RS/6000 SP node and the SP Switch Router is included.

More details about the underlaying Software and Hardware can be found in Section 2.2, “GRF Software” on page 18 and Section 2.3, “GRF Hardware” on page 24.

Section 2.4, “PSSP Enhancements” on page 40 describes the enhancements in the PSSP Software for the support of the dependent node.

Some planning considerations which should be considered can found in Section 2.5, “Planning for the GRF” on page 63 and Section 2.6, “Planning for the Dependent Node” on page 65.

2.1 Overview

The purpose of the SP Switch Router Adapter is to allow the GRF ("goes really fast"), manufactured by Ascend, to forward SP Switch IP traffic to other networks. The GRF was known as the High Performance Gateway Node (HPGN) during the development of the adapter. IBM remarkets models of the GRF that connect to the SP Switch as the SP Switch Router model 04S (9077-04S) and model 16S (9077-16S). These models are not available directly from Ascend.

Note: In the remainder of this book, we refer to the SP Switch Router as the GRF.

The distinguishing feature of the GRF, when compared with other routers, is that it has an SP Switch Router Adapter and therefore can connect directly to the SP Switch (see Figure 1 on page 6).

© Copyright IBM Corp. 1998

5

IBM 9570 Disk

Array

Subsystem

SP Switch

HIPPI

Adapter

Adapter

HIPPI

Adapter

 

 

ATM

 

 

 

OC-12c

ATM

 

 

ATM

Switch

SP

SP Switch

OC-3c

 

Adapter

 

 

 

 

 

Switch

Processor

 

Nodes

8-port

 

E-net 10/100

SP Switch

Adapter

4-port FDDI

SP Switch

Router

SP System

Figure 1. SP Switch Router

The RS/6000 SP software treats this adapter as an extension node. It is a node because it takes up one port in the SP Switch and is assigned a node number. It is described as an extension because it is not a standard RS/6000 SP node, but an adapter card that extends the scope of the RS/6000 SP.

Although the term extension node represents the node appearance of the adapter, it does not define the connection. An extension node adapter is used for that purpose. Each extension node has an extension node adapter to represent its connection to the SP Switch.

2.1.1 Motivation

A thin node, which has a single microchannel, is unable to deliver more than about 30 MB/s to or from the SP Switch. Using a wide node, this number increases to 65 MB/s but is still unable to provide full bandwidth to even one HIPPI interface. It is also unable to feed 4 FDDI or 4 Ethernet 100BaseTx cards at full bandwidth.

A 135 MHz wide node’s CPU becomes saturated at about 5000 packets/second. A 10 Mb/s Ethernet uses a maximum of 1500 bytes for a

6 IBM 9077 SP Switch Router: Get Connected to the SP Switch

packet size. This would only enable a wide node to handle approximately 7.5 MB/s of IP traffic.

Since Ascend’s business depends on keeping pace with networking technology, they already support the major interfaces today. The 9077 will be able to take advantage of any new interfaces that are developed in the future as well, with no further development time or money expended.

With some interfaces requiring up to 5 slots, even a wide node can run out of available slots. This forces additional nodes to be added even if there are no performance limitations in the current configuration.

Since there are no hot plug capabilities with an SP node, any failure means downtime on all interfaces configured in that node, and at times a lengthy maintenance procedure. Redundancy is not built into the SP node’s architecture.

These facts are illustrated in Figure 2:

 

 

 

SP Node

9077

Bus

Shared

Non-blocking Crosspoint Switch

 

 

 

1 MCA per thin node

250 ns path setup

 

 

 

2 MCA per wide node

 

 

 

Route Table

Centralized

Independent lookups per card

 

Cache,Software Based

Hardware based, <2.5 µs

 

Cache hits <50% typical

150,000 route capacity per card

Scalability

Single port per card

High per card port density

 

Single CPU

Per card

 

Limited by shared bus

 

 

Processors

 

 

 

 

 

 

 

 

 

Route Tables

 

 

 

 

 

 

 

 

 

 

 

 

Lookup engines

 

 

 

 

 

 

 

 

 

 

Each card has dedicated bandwidth

Throughput

5000 pps

Up to 130,000 pps

 

30 MB/s per thin node

100 MB/s per card slot, full duplex

 

65 MB/s per wide node

 

 

 

Support

No support for:

Support for:

 

 

 

HSSI

 

 

multiple SP Switch interfaces

 

 

 

 

 

 

 

 

ATM OC12

 

 

High-speed networks such as

 

 

 

 

 

 

 

 

Sonet

 

 

HIPPI

 

 

 

Multiple SP Switch Adapters

 

 

Protocols

 

 

 

 

 

 

 

Figure 2. Functional Comparison

Router Node 7

2.1.2 Design Objectives

Because the dependent node is part of the RS/6000 SP, it had to be packaged and assigned some roles consistent with other RS/6000 SP nodes. Changes were made to the RS/6000 SP to incorporate management requirements for the dependent node.

Ease of design and implementation were important objectives in the design. These were accomplished by limiting the amount of switchcontrol protocol for the dependent node.

New SDR (System Data Repository) classes were created to manage dependent nodes. This was done to minimize the scope of the changes and the exposure to side effects that dependent nodes may cause if they were represented as standard nodes in the SDR.

2.1.3 What is a Router

One of the basic functions of the Internet Protocol (IP) is its ability to connect between different networks. This is due its routing algorithm and its flexibility to use almost any physical network below. A system that connects different physical or logical networks and directs traffic is termed a router, although the older term IP gateway is also used.

Again, IP routing is the passing of an IP packet from one device to another by sending it on a physical or logical interface. routers interconnect networks so that IP traffic can be routed between the systems in the networks, as shown in Figure 3 on page 9.

8 IBM 9077 SP Switch Router: Get Connected to the SP Switch

Network 1

Network 3

Router

Network n

Network 2

Figure 3. Typical Router Configuration

Routers help to reduce the amount of processing required on local systems, since they perform the computation of routes to remote systems. For example, a system can communicate with a remote system by passing the message (or packets) to the router. The router works out how to get to the remote system and forwards the message appropriately.

Storing routes on the system takes up memory. But because a system does not have to store routes to systems not in its own subnet, the route table uses less storage space and thereby frees up memory for other work.

The use of routing reduces network traffic, because routers encourage subnetting, which creates a smaller network of systems. By having smaller networks, network traffic congestion is reduced and overall network performance and traffic control are improved.

A network’s routing configuration does not always require a routing protocol. In situations where the routing information does not change, for example, when there is only one possible route, the system administrator usually builds the routing table manually. Some networks have no access to any other TCP/IP networks, and therefore do not require routing tables at all. The three most common routing configurations are:

Router Node 9

Minimal routing

A network completely isolated from all other TCP/IP networks requires only minimal routing. A minimal routing table is usually built by ifconfig when the network interfaces are configured. If your network does not have direct access to other TCP/IP networks, and if you are not using subnetting, this may be the only routing table you require.

Static routing

A network with a limited number of gateways to other TCP/IP networks can be configured with static routing. When a network has only one gateway, a static route is the best choice. A static routing table is constructed manually by the system administrator using the route command. See Figure 4. Static routing tables do not adjust to network changes, so they work best where routes do not change.

Source Host

 

 

Destination Host

Application

 

 

Application

Transport

Gateway

Transport

Destination

Gateway

Destination

Gateway

Destination

Gateway

192.168.1.0

192.168.12.3

192.168.1.0

192.168.1.5

192.168.1.0

192.168.1.2

192.168.12.0

192.168.12.2

192.168.12.0

192.168.12.3

default

192.168.1.5

default

192.168.12.1

default

192.168.12.1

 

 

Network Access

Network Access

Network Access

192.168.12.2

192.168.12.3

192.168.1.5

192.168.1.2

 

192.168.12.0

 

192.168.1.0

 

Figure 4. Table-Based Routing

Dynamic routing

A network with more than one possible route to the same destination should use dynamic routing. A dynamic routing table is built from the information exchanged by the routing protocols. The protocols are designed to distribute information that dynamically adjusts routes to reflect changing network conditions. Routing protocols handle complex routing

10 IBM 9077 SP Switch Router: Get Connected to the SP Switch

situations more quickly and accurately than a system administrator can do. Routing protocols are designed not only to switch to a backup route when the primary route becomes inoperable; they are also designed to decide which is the "best" route to a destination. On any network where there are multiple paths to the same destination, a dynamic routing protocol should be used.

2.1.4 Routing without the GRF

Before the GRF was available, there were only two ways to get IP traffic from remote systems to reach the RS/6000 SP nodes:

1.By putting an additional IP adapter into every RS/6000 SP node.

2.By designating one or two nodes to act as a router (as shown in Figure 5).

 

Node

Router

 

Internet/Intranet

 

 

 

Node

 

SP Switch

Node

ATM

 

 

. . .

 

 

Node

FDDI

 

 

 

 

Ethernet

Figure 5. Routing without GRF

The first option was usually not chosen because it was too costly for the following reasons:

For systems with a large number of nodes having multiple IP adapters for each RS/6000 SP node can be expensive.

The number of I/O slots in the RS/6000 SP node is limited. In addition, these slots are required to perform other tasks for the system, such as connecting to disk or tape. Using these I/O slots to connect IP adapters restricts the functions of the RS/6000 SP node.

Router Node 11

The second case has proven to be very expensive as well. The RS/6000 SP node was not designed for routing. It is not a cost-effective way to route traffic for the following reasons:

It takes many CPU cycles to process routing. The CPU is not a dedicated router and is very inefficient when used to route IP traffic (this processing can result in usage of up to 90%).

It takes a lot of memory to store route tables. The memory on the RS/6000 SP node is typically more expensive than router memory.

The CPU on a node can only drive the system I/O bus at less than 80 megabytes per second, which is less than what a high-end router can do.

For these reasons, the performance of routers in handling IP traffic from remote systems to the RS/6000 SP nodes was limited.

2.1.5 Routing with the GRF

The GRF is a dedicated, high-performance router (see Figure 6). Each SP Switch Router adapter can route up to 30,000 packets per second and up to 100 MB/s into the SP Switch network in each direction simultaneously.

Node

 

Internet/Intranet

Node

 

 

. . .

 

 

SP Switch

 

 

Node

 

ATM

 

 

 

GRF

FDDI

 

 

 

 

Ethernet

Figure 6. Routing with GRF

12 IBM 9077 SP Switch Router: Get Connected to the SP Switch

The GRF uses a crosspoint switch (see Figure 7) instead of an I/O bus to interconnect its adapters. This switch is capable of 4 or 16 Gbit/s (model dependent) and gives better performance than the MCA bus.

IP Switch Control Board

 

 

 

Route

 

4Gb/s

Manager

 

Crosspoint

 

 

Switch

 

 

1 Gb/s to each Media Card

 

Switch Engine Interface

T3-OC12

 

 

 

Route

I/O

IP

LAN/WAN

Table and

Buffering

Packet

 

Lookup

 

Forwarding

 

LAN/WAN Interfaces

 

 

Media Cards

 

Figure 7. GRF 400

In conventional routers, each packet is processed at each gateway (also called hop) along a path. The processing is done at the Layer 3 level (see Figure 8 on page 14) and requires a router’s CPU to process both the packet and the route information. Conventional routers use shared resources, which leads to congestion and poor scalability and performance. Software-based route-table lookups can be very slow, if the route-table is not in cache.

Router Node 13

2

5

Router

Unswitched Data

Switched Data

1

 

4

 

Router

 

3

 

 

 

 

Router

Disadvantages:

Shared data paths

All processing done on Layer 3

Slow softwarebased processing

Layer 3

Layer 2

 

 

 

Router

Router

 

 

 

 

 

 

Process

 

 

Process

 

 

 

 

 

 

......Packets

 

 

......

 

 

 

 

 

 

 

 

Packets

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

.........

 

....

............

....

.....

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure 8. Conventional Routers

The SP Switch Router provides near wire-speed packet forwarding while using standard routing protocols. This ensures interoperability with other network technologies and does not require a specific network architecture, such as ATM. It works equally well in large and small networks. At each hop where a routing switch is used, routes are processed at Layer 3 but the packet is forwarded at Layer 2 (see Figure 9 on page 15). In the case of the SP Router, the route processing is done through hardware, so all processing is done at near-wire speed.

14 IBM 9077 SP Switch Router: Get Connected to the SP Switch

 

Unswitched Data

 

Switched Data

2

5

 

 

 

Router

Examples:

 

1

 

4

Ascend GRF, SP Switch Router

 

 

Router

Cisco 12000

 

 

3

 

 

 

 

 

 

 

 

 

 

Router

 

 

 

 

 

Disadvantages:

 

 

 

 

Hardware can be hard to upgrade *

 

 

 

 

Reduced routing functions *

 

Switch

 

Switch

 

Layer 3

Process Route

 

Process Route

Advantages:

 

 

 

 

Behave like traditional router

Layer 2 ..... ..... ......... ..... .....

Not dependent on a network architecture

Interoperability

Figure 9. Switched Routers

Other advantages of using GRF are as follows:

Availability of a redundant power supply

Availability of a redundant fan

Availability of a hot-swappable power supply

Availability of a hot-swappable fan (model 16S only)

Availability of hot-swappable media adapters (to connect to networks)

Scalability of up to 4 or 16 media adapters, depending on the GRF model

Perhaps the greatest advantage of using the GRF is improved price/performance. As previously mentioned, the GRF is a dedicated router, and as such it is much more cost effective for routing IP traffic than using dedicated RS/6000 SP node.

2.1.6 Overview of Supported Routing Protocols

In addition to static routes, various routing protocols are available on the GRF, as follows:

RIPRouting Information Protocol Version 1 or 2 (RIP 1 or 2)

OSPFOpen Shortest Path First

IS-ISIntermediate System to Intermediate System (an OSI gateway protocol)

Router Node 15

MulticastIP Multicast and OSPF Multicast

EGPExterior Gateway Protocol

BGPBorder Gateway Protocol Version 3 or 4 (BGP 3 or 4)

More details about the various protocols are in Section 2.2.2, “Supported Routing Protocols” on page 20.

2.1.7 Media Adapters At-a-Glance

Available IP forwarding media cards are:

1-port 100 Mbyte/s Switch Adapter

8-port 10/100 Mbit/s Ethernet cards

2-port 155 Mbit/s OC-3 ATM (Asynchronous Transfer Mode UNI 3.0/3.1)

1-port 622 Mbit/s OC-12 ATM

4-port 100 Mbit/s FDDI (Fiber Distributed Data Interface)

1-port 800 Mbit/s HIPPI (High Performance Parallel Interface)

2-port 52 Mbit/s or 45 Mbit/s DS-3 HSSI (High Speed Serial Interface)

1-port 155 Mbit/s IP/SONET OC-3c

More details are in Section 2.3.8, “Other Media Cards” on page 39.

2.1.8 Benefits of the GRF

The crosspoint switch is a nonblocking crossbar. This architecture is faster than an RS/6000 SP node, in which media adapters communicate through a shared microchannel bus.

To take advantage of the fast I/O provided by the crosspoint switch, fast route table access time is required. The GRF can store up to 150,000 routes in memory on each media card, while an RS/6000 SP node can store only hundreds. It is said that you need about 50,000 routes for the whole Internet. This means that the GRF is able to retrieve a route faster than an RS/6000 SP node.

The GRF is able to route up to 2.8 million packets per second for the 4-slot model and 10 million packets per second for the 16-slot model.

All the media adapters on the GRF are hot-pluggable. This differs from using an RS/6000 SP node as your router. Should any network adapter on the RS/6000 SP node fail, the node has to be brought down to replace the faulty

16 IBM 9077 SP Switch Router: Get Connected to the SP Switch

adapter. As a result, other network adapters are brought down as well. Bringing down the router will impact all the networks in the location.

Each RS/6000 SP is allowed to connect to multiple SP Switch Router Adapters, and it does not matter if these adapters are on different GRFs. Connecting multiple SP Switch Router adapters to either different partitions in an RS/6000 SP or to different RS/6000 SPs allows them to communicate with each other and with the other GRF media adapters via the SP Switch.

Attention

The SP Switch Router model 04S can support four media cards such as FDDI or ATM. The SP Switch Router model 16S can support 16. All card slots could be occupied by SP Switch Router adapters; this means a maximum of 4 SP Switch Router adapters for model 04S and a maximum of 16 SP Switch Router adapters for model 16S.

Note: The number of packets that the GRF can route per second depends on the following:

The type of media adapter

The size of the packet

2.1.9 Price Comparison

Figure 10 on page 18 shows a price comparison between an RS/6000 SP node solution and a GRF based solution for three sample configurations.

Router Node 17

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

9077-04S with one SP Switch Router Adapter

 

$ 53,000

 

 

 

 

 

 

 

HIPPI Adapter

 

 

 

13,500

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

9077-04S

 

 

 

 

 

 

 

 

 

 

 

 

 

with one SP Switch Router Adapter

 

 

66,500

$ 53,000

 

 

 

 

 

 

135 MHz

HIPPI A apter

 

 

 

48,000

 

 

 

 

 

 

 

 

 

19,000

 

 

 

 

 

 

Wide Node

9077-04S

 

 

 

3,200

 

 

 

 

 

 

64 MB memory

with one SP Switch Router Adapter

 

72,000

 

$ 53,000

 

 

 

 

4.5 GB Disk

 

1,950

 

 

 

 

 

HIPPI Adapter

 

 

 

 

 

 

 

 

 

48,000

 

 

 

 

 

Ethernet 135 MHz Wide Node

 

 

595

 

20,000

 

 

 

 

 

64 MB memory

 

 

10,000

3,200

 

 

 

 

 

 

 

 

 

 

 

 

SP Switch Adapter

 

 

 

 

17,500

1,950

 

73,000

 

 

 

 

 

4.5 GB Disk

 

 

 

 

 

 

 

 

 

HIPPI Adapter

135 MHz Wide Node

 

 

 

 

595

 

48,000

 

 

 

 

 

Ethernet

 

 

 

81,245

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

SP Switch

Adapter64 MB memory

 

 

 

10,000

 

3,200

 

 

 

 

 

4 FDDI 1-port SAS4.5GBAdaptersDisk *

 

 

 

 

15,980

 

1,950

 

 

 

 

 

 

 

 

 

 

 

 

 

595

 

 

 

 

 

 

Ethernet

 

 

 

 

75,730

 

 

 

 

 

 

 

 

 

 

 

 

10,000

 

 

 

 

 

 

SP Switch Adapter

 

 

 

 

 

 

 

 

 

 

 

 

2 ATM 155 Adapters *

 

 

 

 

 

5,390

 

 

 

 

 

 

 

 

 

 

 

 

 

69,135

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure 10. Price Comparison

These price comparisons are based on US prices as of March 1998. In other countries these prices may be different. The basic message of these charts is that the solutions based on the GRF could be quite competitive and will quite often be cheaper than the conventional configurations.

Let us look, for example for a solution connecting an RS/6000 SP via HIPPI to a mainframe system. The first chart shows that a GRF solution is cheaper than adding a dedicated node for the HIPPI connection to your system, apart from the fact that the GRF solution is the better choice from a performance point of view.

It is nearly the same if you need a connection to an FDDI network. One GRF FDDI media card offers four independent singlering connections. An offer based on an dedicated SP node is more expensive than a GRF solution.

Our example on the third chart focuses on an ATM connection. In this case, the RS/6000 SP node based solution is the more reasonable solution, if you consider only the price. But the difference is not that much; the growth path with the GRF-based offer will be better than the node solution.

2.2 GRF Software

The software functionality of the GRF is distributed between the Router Manager on the IP Switch Control Board (see also Section 2.3.2, “GRF Features” on page 26) and the individual media cards. While the Route

18 IBM 9077 SP Switch Router: Get Connected to the SP Switch

Manager updates the system routing tables and performs other administrative functions, the intelligent processors on each media card perform all routing functions. This design supports efficient distributed processing of router operations.

2.2.1 IP Protocol

The GRF supports IP datagram routing between major types of standard media. The implementation conforms with IP Version 4 and routing specifications described in Internet RFCs.

Each media card has a complete set of route and Address Resolution Protocol (ARP) information contained in the program memory space of the card’s on-board processor. IP packets are buffered in large transmit and receive buffers from which they are transmitted across the central switch fabric to the destination media. Any difference in MTU size (large MTU to smaller MTU) is handled by packet fragmentation as specified in the IP standard. Logic on the destination media is responsible for any media-specific processing of the packet, such as producing 53-byte cells for ATM.

Data Forwarding

Individual media cards maintain their own route tables, perform lookups, and autonomously handle the passing of datagrams to other media cards for export, without intervention of the Router Manager. Layer-3 decisions are local to each card.

Route Table Implementation

Critical to providing sustained performance in a highly dynamic environment are the cacheless route table and route lookup implementation. Each card carries a complete copy of the route table and can support up to 150,000 entries.

Keeping pace with significant advances in routing, the GRF also supports variable-length subnet masking and route aggregation.

Router Node 19

Subnet Masking/Supernetting

Variable length subnet masking is a classless addressing scheme for interdomain IP packet routing. It is a way to more efficiently manage the current 32-bit IP addressing method. Subnet masks let sites configure the size of their subnets based on the site needs, not on the arbitrary Class A, B, and C structure originally used in the Internet addressing. Class-based addressing restricts the boundary to the 8-bit boundaries and is implicitly derived from the first eight bits of the network address. The new addressing method allows the network portion of an IP address to be separated from the host portion of the address at any point within the 32-bit address space. This expanded boundary is called the "netmask" and is explicitly provided to the router along with the network address information. Class-based addressing restricts the boundary to the 8-bit boundaries and is implicitly derived from the first eight bits of the network address.

Subnet masking offers a number of benefits by extending the current address space. By eliminating implicit netmask assignments, addresses can now be assigned from any unused portion of the entire 32-bit address range rather than from within a specific subset of the space previously called a class. Since it hides multiple subnets under a single network address, this method is called supernetting.

Classless addressing allows the network administrator to further apportion an assigned address block into smaller network (or host) segments based on powers of two (2, 4, 8, 16 networks, for example). Knowledge of the apportioned segments need not be communicated to exterior peers. They need only a single pointer to the entire address block. Not only does subnet masking better utilize address space, but implemented properly it results in significantly smaller routing tables.

2.2.2 Supported Routing Protocols

In the days of a single Internet, core groups of independent networks were called autonomous systems. We will use the term autonomous systems (AS) in the following description of protocols. The routing protocols supported on the GRF can be divided into two classes: Interior routing protocols or interior gateway protocols (IGPs) and Exterior routing protocols (EGPs).

Interior routing protocols

Interior routing protocols are used to exchange routing information between routers within a single autonomous system. They are also used by routers that run exterior protocols to collect network reachability

20 IBM 9077 SP Switch Router: Get Connected to the SP Switch

information for the autonomous system. Here is the list of interior protocols supported by the GRF:

RIP

The Routing Information Protocol (RIP), as delivered with most UNIX systems, is run by the routing deamon routed. During the startup of routed a request for routing updates is issued. After that, the daemon listens for responses to the request. Systems that are configured to supply RIP information hear this request and respond with update packets based on the information in the system’s routing table. The update packets contain the destination addresses from the routing table and the routing metrics associated with each destination. Update packets are send on request and periodically to keep routing information accurate.

OSPF

Open Shortest Path First (OSPF) is defined by RFC 2178 (Request for Comments). It is a link-state protocol and very different from RIP. A router running RIP shares information about the entire network with its neighbors. A router running OSPF shares information about its neighbors with the entire network. The "entire network" means, at most, a single autonomous system. OSPF further defines a hierarchy of routing areas within an autonomous system:

Areas

These are sets of networks within a single autonomous system that have been grouped together. The topology of an area is hidden from the rest of the autonomous system, and each area has a separate topology database. Routing within the autonomous system takes place on two levels, depending on whether the source and destination of a packet reside in the same area (intra-area routing) or different areas (inter-area routing).

Intra-area routing is determined only by an area’s own topology. That is, the packet is routed solely on information obtained within the area.

Inter-area routing is always done via a backbone.

The dividing of an autonomous system into areas enables a significant reduction in the volume of routing traffic required to manage the routing database for a large autonomous system.

Backbone

The backbone consists of those networks not contained in any area, their attached routers, and routers that belong to multiple areas.

Router Node 21

Every area must connect to the backbone, because the backbone is responsible for the distribution of routing information between areas. The backbone itself has all the properties of an area. Its topology is separate from that of other areas.

Subarea

A subarea has only one area border router, which means that there is only one route out of the area. In this case, the area border router does not need to advertise external routes to the other routers within the subarea. It can simply advertise itself as the default route.

The sequence of operations performed by OSPF routers is as follows:

1.Discovering OSPF neighbors

2.Electing the Designated Router

3.Forming adjacencies

4.Synchronizing databases

5.Calculating the routing table

6.Advertising Link States

Routers go through these steps when they first come up, and repeat them in response to the events that occur in the network. Each router must perform each of these steps for each network it is connected to, except for the calculation of the routing table. Each router generates and maintains a single routing table for all networks.

Multicast

IP Multicast

The GRF supports IP multicast routing per RFC 1112 and some components of RFCs 1301 and 1469. The implementation includes the IP multicast kernel modifications, dynamic route support and mrouted (multicast route daemon).

Data that arrives at a GRF interface is duplicated and forwarded to multiple destination interfaces. The multicast packet’s destination address is a Class D address. A lookup of the multicast route table returns a list of Virtual Interfaces (VIFs) to which the packet is sent.

OSPF Multicast

The GRF uses the multicast capability of OSPF Version 2, as described in RFC 1583 and RIP Version 2, for communications between routers.

22 IBM 9077 SP Switch Router: Get Connected to the SP Switch

Host extensions for IP multicasting as described in RFC 1112 are also provided. The Router Manager acts as a host and uses the

Internet Group Management Protocol (IGMP), version 2, to add and delete its membership in multicast groups. Accordingly, the Route Manager "joins" the appropriate routing protocol host groups for OSPF and RIP.

IS-IS

Intermediate System to Intermediate System (an OSI gateway protocol) is a protocol similar to OSPF: it also uses a Link State, Shortest Path First algorithm. However, IS-IS is an OSI protocol used for routing Connectionless Network Protocol (CLNP) packets within a routing domain. CLNP is the OSI protocol most comparable to IP.

Exterior Routing Protocols

Exterior Routing Protocols are used to exchange routing information between routers in different autonomous systems. Here is the list of exterior routing protocols supported by the GRF:

EGP

The Exterior Gateway Protocol (EGP) is the protocol used for exchange of routing information between exterior gateways (not belonging to the same autonomous system).

EGP gateways may only forward reachability information for networks within their autonomous system. This routing information must be collected by the EGP gateway, usually via an GP).

BGP

Border Gateway Protocol (BGP) is the leading exterior routing protocol of the Internet and is replacing EGP as the exterior protocol of choice. It is based on the OSI InterDomain Routing Protocol (IDRP). BGP supports policy-based routing, which uses nontechnical reasons (for example, organizational, political, or security considerations) to make routing decisions. Thus, BGP enhances an autonomous system’s ability to choose between routes and to implement routing policies without relying on a central routing authority.

2.2.3 Filtering

IP filtering supports specific permit or deny decisions for each instance of a filter (per logical interface). The criteria within each filter may include any combination of the following:

• Protocol (ICMP, TCP, UDP)

Router Node 23

Source address

Destination address

Protocol port number (single number, or range, or ranges) for TCP and UDP

Established TCP connections

2.2.4 System Management

The GRF currently supports the Simple Network Management Protocol

(SNMP) Version 1, which provides a mechanism for remote query or setting operational parameters for the device. Media cards collect network statistics, which can be reported to network management packages via the Router Manager on the IP Switch Control Board. In addition to collection statistics and other management information, the SNMP agent is also capable of issuing traps. For more information, see Section 2.4.5, “SP Extension Node SNMP Manager” on page 58.

2.3 GRF Hardware

As already mentioned, the unique GRF switching architecture is specially designed for high-performance packet forwarding. The following sections give you more details about the various hardware components.

2.3.1 GRF Block Diagram

Figure 11 on page 25 shows the two models: the 4-slot and the 16-slot model.

24 IBM 9077 SP Switch Router: Get Connected to the SP Switch

 

 

 

 

 

 

 

 

 

 

19"

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Cooling Fan Drawer

 

 

 

 

 

 

21"

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

19"

Forwarding Media Card

Forwarding Media Card

Forwarding Media Card

Forwarding Media Card

Forwarding Media Card

Forwarding Media Card

Forwarding Media Card

Forwarding Media Card

Control Board IP Switch Board Forwarding Media Card

Forwarding Media Card

Forwarding Media Card

Forwarding Media Card

Forwarding Media Card

Forwarding Media Card

Forwarding Media Card

Forwarding Media Card

Power

IP Switch Control Board

Supply

IP Forwarding Media Card

IP

IP

IP

IP

IP

IP

IP

IP

IP

IP

IP

IP

IP

IP

IP

IP

 

IP Forwarding Media Card

Power

 

 

 

 

Power

 

 

 

Power

 

 

 

IP Forwarding Media Card

 

 

 

 

 

 

 

 

 

 

Supply

IP Forwarding Media Card

 

 

 

 

Supply

 

 

 

Supply

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

5.25"

GRF 400

 

 

 

 

 

 

 

 

GRF 1600

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure 11. GRF Models

The SP Switch Router model 9077 04S (GRF 400) can accommodate up to four media adapters.

The SP Switch Router model 9077 16S (GRF 1600) can accommodate up to 16 media adapters.

Each adapter enables the GRF to connect to one or more networks.

Each of the models has an additional slot for the IP Switch Control Board, which is used to control the router.

GRF 400

PartDescription

Cooling FansThese are located on the right side of the chassis and cannot be accessed without bringing down the GRF. The fans are redundant, allowing service to be deferred until it is convenient to bring down the GRF.

Media CardsThere are four media card slots on this chassis. They are slotted horizontally and are located at the bottom of the chassis.

IP Switch Control BoardThis board is located at the top of the four media slots and is also slotted horizontally.

Router Node 25

Power SupplyThe left side of the chassis is reserved for the two power supplies that are required for redundancy.

The failed power supply can be hot-swapped out of the GRF chassis.

GRF 1600

Part Description

Cooling FansThese are located at the top of the chassis, and can be accessed separately from the other parts of the GRF. The fan tray contains redundant fans and is hot-swappable.

Media CardsThere are 16 media card slots on this chassis. They are slotted vertically. Eight of the cards are on the left side of the chassis, and eight are on the right side.

IP Switch and

Control BoardThese boards are located in the middle of the 16 media slots and are also slotted vertically.

Power SupplyThe base of the chassis is reserved for the two power supplies that are required for redundancy. The failed power supply can be hot-swapped out of the GRF chassis.

2.3.2 GRF Features

GRF has the following features:

Redundant power supply

Should any power supply fail, a message is sent to the control board. The power supply will automatically reduce its output voltage if the temperature exceeds 90 °C (194 °F). If the voltage falls below 180V, the GRF automatically shuts down.

Hot-swappable power supply

The faulty power supply can be replaced while the GRF is in operation.

Redundant fan

For the GRF 1600 model, if one fan breaks down, a message is sent to the control board.

For both models, when the temperature reaches 53°C (128 °F), an audible alarm sounds continuously, and a message is sent to the console and logged into the message log.

26 IBM 9077 SP Switch Router: Get Connected to the SP Switch

If the temperature exceeds 57.5 °C (137 °F), the GRF does an automatic system shutdown.

Hot-swappable fan

For the GRF 16S model, the cooling fan can be replaced while the GRF is in operation.

Hot-swappable adapters

There are two types of adapters on the GRF: the media adapters and the IP Switch Control Board.

The media adapters are independent of each other and can be replaced or removed without affecting any other adapter or the operation of the GRF.

However, the IP Switch Control Board is critical to the GRF. Should this board be unavailable, the router fails.

Crosspoint switch

The crosspoint switch is a 16x16 (16Gb per second) or 4x4 (4Gb per second) crossbar switch for the GRF 16S and GRF 04S, respectively, see Figure 12 on page 27. It is the I/O path used when the media adapters need to communicate with each other.

Serial Interface

 

 

 

 

r

QBRT

RT

 

 

r

 

 

 

 

fe

 

 

fe

 

 

 

f

 

 

 

f

 

 

B

u

 

 

 

 

u

 

T

x

 

 

 

 

x

B

 

 

 

 

CPU

 

 

 

 

 

 

 

 

R

 

 

 

 

 

 

Media Interface

 

 

 

 

 

 

 

 

 

ATM OC-3c

MM

 

 

 

 

 

 

 

 

IP/SONET OC-3c

MM

 

 

 

CPU

............

..........

Combus

80 Mb/s bus

User Interface

Out-of-Band

 

Dynamic Routing Engine (Route Manager)

Configuration/Control

16MB Rx / 16MB Tx Buffers

Speed decoupling

WAN delays

QBRT Route Table Lookup

Times range from 1-2.5 µs

On-Board Processor

IP Header

Route decisions

4 Gb/s Switch

Figure 12. GRF Architecture

Some parts in Figure 12 need to be explained:

Router Node 27

1.Normally, all media cards have a 4 MB send buffer and a 4 MB receive buffer, except the SP Switch Adapter card, which has a 16 MB buffer size for each buffer. See also Section 2.3.5, “Characteristics of GRF Media Cards” on page 36 and Figure 19 on page 37.

2.Quick Branch Routing Technology (QBRT) is a hardware-assisted route table lookup. Route lookup times range from 1 - 2.5 µs with up to 150,000 next-hop routes in the table. Not all media cards use QBRT. Cards that do not use QBRT use a microcode lookup.

The benefit of this architecture is that the entire route table can be stored locally on the media card and searched quickly. In the traditional cached route table method, a small number of routes can be stored and searched locally. However, when a large number of routes is desired, or the kind of traffic one would see on the Internet backbone arises, caching is inadequate. Inevitably, cache misses occur, and route table lookups are performed at a limited, central, shared resource.

Performance is enhanced even further with parallel processing of table lookups occurring on each media card, which is another technique that helps assure linear scalability. The router manager on the controller board, which also contains the switch fabric, maintains the master route table and distributes updates simultaneously to all installed media cards, even as the cards continue their forwarding functions.

3.On-Board Processor

4.Router management takes place on the IP Switch Control Board (see Section 2.3.3.2, “IP Switch Control Board Components” on page 33), based on a 166 Mhz Pentium processor. It is responsible for system monitoring, configuration management and the user interface.

5.The GRF communications bus (Combus) is an "out-of-band" data path for configuration, control and monitoring of media cards. The Combus connects the IP Switch Control Board to the media cards independent of the switch connection to each card. It is not used for routed data between the cards. Route update packets received on any media card are also sent across the Combus to the Route Manager and, therefore, do not have to compete with normal IP traffic. The Combus is a serial bus with a transferrate of 80 Mb/s and is FIFO-buffered. The Arbitration Logic is on the IP Switch Control Board.

2.3.2.1 Data Packet Processing

With the knowledge about the local routing functions of the media cards, we now look at Figure 13 on page 29 to see how a data packet is transferred from one media card to another.

28 IBM 9077 SP Switch Router: Get Connected to the SP Switch

 

External

 

 

External

 

 

Interface

 

 

Interface

 

Buffer

DMA

Buffer

Buffer

DMA

Buffer

CPU

CPU

QBRT

QBRT

Tx

Rx

Tx

Rx

DMA

DMA

 

 

 

 

 

Internal

 

 

Internal

Switch

 

 

 

Control Board

 

Interface

 

 

Interface

 

 

 

 

 

 

Dynamic

 

 

 

 

 

Route

 

 

 

 

 

Manager

Figure 13. Data Packet Transfer

The routing can be divided into the following steps:

1.A data packet is received by the media card.

2.The packet is transferred to the receive buffer by the DMA engine.

3.The CPU examines the header and gives the destination address to the route lookup hardware.

4.The QBRT finds the next hop in the route table.

5.A special header is added to the packet by the CPU. This header contains information for the downstream card to process the packet, as well as the next hop found by the QBRT.

6.The packet is then transferred to the internal serial interface by the DMA engine.

7.A connection to the downstream card is set up through the switch.

8.The serial stream is converted back to parallel format by the downstream card.

9.The packet is transferred to the transmit buffer by the DMA engine.

Router Node 29

10.The header is examined by the CPU, which uses the information to build a new header that will deliver the data across the media interface.

11.The DMA engine transfers the packet to the media interface.

12.The packet is transferred across the media.

2.3.2.2 Routing Packet Processing

The processing of packets with routing information is a little bit different from the data packet processing procedure as you can see in Figure 14.

 

External

 

 

External

 

 

Interface

 

 

Interface

 

 

DMA

 

 

DMA

 

Buffer

CPU

Buffer

Buffer

CPU

Buffer

QBRT

QBRT

Rx

Tx

Rx

Tx

DMA

DMA

 

 

 

 

 

Internal

 

 

Internal

Switch

 

 

 

Control Board

 

Interface

 

 

Interface

 

 

 

 

 

 

Dynamic

 

 

 

 

 

Route

 

 

 

 

 

Manager

Figure 14. Routing Packet Processing

These are the steps for processing routing packets:

1.A routing packet is received by the media card.

2.The packet is transferred to the receive buffer by the DMA engine.

3.The CPU examines the header and gives the destination address to the route lookup hardware.

4.The QBRT finds the next hop in the route table.

5.A special header is added to the packet by the CPU. This header contains information for the downstream card. The result of the hardware lookup determines whether the packet should be forwarded to the Router Manager.

30 IBM 9077 SP Switch Router: Get Connected to the SP Switch

6.The packet is then transferred to the Combus interface by the DMA engine.

7.The packet is sent to the IP Switch Control Board’s Router Manager across the Combus.

8.The Route Manager receives the packet and passes it to the dynamic routing software.

9.The packet is processed and global routing information is determined. 10.Route updates are broadcast across the Combus to all media cards

simultaneously.

11.Each card receives the update packet and makes changes to its route tables.

12.The packet is transferred across the media.

To ensure that dynamic routing packets are not dropped during times of heavy congestion, precedence features are used. Routing packets are given a high-priority tag and a user-configurable threshold for Tx buffers is maintained for high-priority traffic.

2.3.3 IP Switch and Control Board

The control board, also known as the IP Switch Control Board, is accessed through Telnet or a locally attached VT100 terminal. The IP Switch Control Board is supplied with the GRF and is necessary for its operation. The VT100 terminal is not supplied with the GRF. It is only needed for the installation of the GRF.

Using terminal emulation software instead of looking for a real VT 100 terminal may be an alternative. You can use your Control Workstation or one of your SP nodes. Install the ATE package (advanced terminal emulation) on your RS/6000 and establish a serial connection between the system and the router.

After installation, all future access to the GRF can be through Telnet to the IP Switch Control Board’s administrative Ethernet.

The IP Switch Control Board is identified as slot 66 in both GRF models. A sideview of the GRF 400 slot numbering scheme is shown in Figure 15 on page 32.

Router Node 31

Backplane

66

3

2

1

0

IP Switch Control Board

Slot numbers in decimal for media cards

Figure 15. Side View of GRF 400 Chassis with Slots Numbered

The GRF 1600 has 16 media slots. The control board is located in slot 66, as shown in Figure 16.

Backplane

 

 

 

Media cards

 

 

 

Slot numbers

0 1 2 3 4 5 6 7

66

8 9 10 11 12 13 14 15

 

Control board

 

Switch board

Figure 16. Top View of the GRF 1600 Chassis

The CPU in the IP Switch Control Board is a 166MHz Pentium processor and runs a variant of BSD UNIX as its operating system. For this reason, the GRF administrator is assumed to be proficient in UNIX.

The IP Switch Control Board is used to install, boot, and configure the router and its media adapters.

It is also used for the logging of messages, the dumping of memory and status, and to perform diagnostic checking of both the GRF and the media adapters.

32 IBM 9077 SP Switch Router: Get Connected to the SP Switch

2.3.3.1 Route Manager

As already mentioned, the router management takes place on the IP Switch Control Board.

Specific functions of the Route Manager are:

It processes all dynamic routing packets.

It synchronizes the route tables on the media cards.

It controls the media cards: issues interrupts and resets to individual media cards and downloads executable programs and connection information.

2.3.3.2 IP Switch Control Board Components

Let us examine the IP Switch Control Board in more detail.

Figure 17 shows all the components of the IP Switch Control Board:

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Flash

 

 

 

 

32MB

 

32MB

 

 

 

32MB

 

 

32MB

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Memory

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

(85 MB)

 

 

 

 

32MB

 

32MB

 

 

 

32MB

 

 

32MB

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

System Bus

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

CPU

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Pentium

 

 

 

 

 

 

 

 

 

 

 

 

 

PCMCIA

PCMCIA

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

166MHz

 

 

 

 

 

 

 

 

 

 

 

 

 

Card

 

Card

 

 

 

 

 

 

Admin Ethernet

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

(de0)

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure 17. IP Switch Control Board

 

 

 

 

 

 

 

 

 

 

Router Node 33

ItemDescription

MemoryThe IP Switch Control Board comes standard with 128 MB of memory (the four shaded blocks of 32 MB of memory in the upper left corner).

The memory can be upgraded to 256 MB, in increments of 64 MB (the four white blocks of memory).

The system uses the first 32 MB of memory for file system storage. The top half is used for applications such as the SNMP agent, the gated daemon, and for the operating system.

Flash memoryThis memory (the 85 MB ATA flash memory on the system) is used to store the operating system information and the configuration information for the GRF.

System busThis bus is used by the IP Switch Control Board components to communicate with each other.

Pentium processorThis 166 MHz processor drives the IP Switch Control Board and the GRF. As previously mentioned, this processor runs a variant of BSD UNIX, and so it is useful for the GRF administrator to have UNIX management skills.

Administrative EthernetThis Ethernet is known to the GRF as de0. This port supports the 10BaseT or the 100BaseT Ethernets and switches between them automatically, depending on the type of network used.

To use 10Base2 or 10Base5, the user must add a transceiver (supplied by the user).

PCMCIA cardsThe two white blocks at the bottom right corner of the figure are PCMCIA slots.

There are two types of PCMCIA cards:

• Slot A on the 9077 is configured with a 520 MB disk drive. This disk is used to hold the log of the GRF. It may also be used to backup the configuration of the GRF. Making a backup is strongly recommended.

• The PCMCIA modem card, also available as an optional device, allows the user to dial into the GRF through a modem to administer it remotely.

Note: For the initial setup, the console must be available locally, not through the modem.

34 IBM 9077 SP Switch Router: Get Connected to the SP Switch

Additionally, the RS232 port (which is not shown in the figure) allows you to connect the VT100 console by using an RS232 null modem cable. The console and cable must be supplied by the user.

2.3.4 Memory Guidelines for the IP Switch Control Board

As already mentioned, the GRF base system comes with 128 MB of memory. In all GRF memory configurations, 32 MB are used for the file system and the remainder is used for system operations. For example, in the base system, there are 128 MB of total memory, 32 MB of available memory and 78 MB of usable memory. (Refer to Table 1 on page 35 for detailed memory configuration information). Up to six additional 32 MB DRAM SIMMS may be added to support larger dynamic routing tables and larger numbers of peers (for a total of 256 MB, 204 MB usable).

The following table provides guidelines for memory configuration. All media cards can hold up to 150 KB route entries. The control board, depending on memory configuration, can hold 35,805 to 521,730 route prefixes. Select the amount of memory according to your routing environment. Additional memory may be required for higher average numbers of routes per BGP peer.

Table 1.

Memory Configuration

 

 

 

 

 

Customer

Total

Avail.

Usable

No. of

No. of

No. of

Profile

Mem.

 

 

Route

Route

Peer

 

 

 

 

 

Entries

Prefixes

Sess.

 

 

 

 

 

on

in

 

 

 

 

 

 

Media

Dynamic

 

 

 

 

 

 

Cards

Database

 

 

 

 

 

 

 

 

Static Routing

64 MB

32 MB

14 MB

150 KB

35805

0

 

 

 

 

 

 

 

Small POP

128 MB

96 MB

78 MB

150 KB

199485

3

 

 

 

 

 

 

 

Medium POP/

192 MB

160 MB

142 MB

150 KB

362165

7

ISP Backbone

 

 

 

 

 

 

 

 

 

 

 

 

 

Large POP/

256 MB

224 MB

204 MB

150 KB

521730

10

Exchange Point/

 

 

 

 

 

 

Route

Reflection

 

 

 

 

 

 

Server

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure 18 gives an overview of the memory layout and the possible memory extensions.

Router Node 35

64MB RAM

64MB RAM

64MB RAM

- System software

64MB RAM

- Config files

32MB

- Gated binary

(fixed size)

20MB

- Route table

 

 

 

 

 

 

 

 

 

 

- Kernel runs

 

 

 

 

8-12MB

 

 

 

 

(fixed size)

 

= expandable area of RAM

- Gated runs

 

 

Figure 18. System RAM

2.3.5 Characteristics of GRF Media Cards

All GRF media cards (media adapters) are self-contained and independent of other media adapters.

Each media card has an onboard processor that is responsible for IP forwarding on the media adapter.

Each media card has two independent memory buffers, a 4 MB send buffer and a 4 MB receive buffer. These buffers are necessary to balance the speed differences between the media adapters, because they have different transfer rates.

Each onboard processor has local memory that can contain a local route table with up to 150,000 entries, to be used for routing on the media adapter. Because these route entries are in local memory, access to them is very fast. When the media adapter is started up, it gets its initial route entries from the IP Switch Control Board.

2.3.6 SP Switch Router Adapter

The GRF supports a number of media adapters. Figure 19 describes the SP Switch Router Adapter in detail. This adapter allows the GRF to connect directly into the SP Switch.

36 IBM 9077 SP Switch Router: Get Connected to the SP Switch

 

 

 

Crosspoint

 

 

 

 

Switch

 

 

 

Media Board

 

 

 

 

Receive

16MB

FIFO

 

 

 

TBIC

Buffer(1)

 

7

3

(1)

 

 

 

 

Receive

 

 

 

 

2

 

 

 

6

Proc & C

Send

 

 

 

 

 

 

5

1

 

Proc & C

 

 

4

0

Send

16MB

FIFO

Serial

(2)

Daughter

 

 

TBIC

Buffer(2)

 

 

Card

SP Switch

 

 

 

 

 

 

 

 

Figure 19. SP Switch Router Adapter

The SP Switch Router Adapter is made up of two parts: the media board and a serial daughter card.

The serial daughter card is an interface for the media board into the crosspoint switch. This switch is the medium by which the GRF (media) adapters talk to each other.

The purpose of the media board is to route IP packets to their intended destination through the GRF. The SP Switch Router adapter described here is used for routing IP packets to and from the SP Switch to other systems connected directly or indirectly to the GRF. A brief description of the components on the media board follows.

Receive TBICThis component receives data segments from the SP Switch and notifies the Receive Controller and Processor that there is data to be transferred to the buffer.

Router Node 37

Receive Controller

and ProcessorThis component recognizes the SP Switch segments and assembles them into IP packets in the 16 MB buffer. Up to 256 IP datagrams can be handled simultaneously. When a complete IP packet has been received, the Receive Controller sends the packet to the FIFO (1) queue for transfer to the serial daughter card.

Buffer (1)This component is segmented into 256 64 KB IP packet buffers. It is used to reassemble IP packets before sending them to the FIFO queue, as switch data segments may arrive out of order and interleaved with segments belonging to different IP packets.

FIFO (1)This component is used to transfer complete IP packets to the serial daughter card and even the flow of data between the SP and GRF crosspoint switch.

FIFO (2)This component receives IP packets from the serial daughter card and transfers them to Buffer (2).

Buffer (2)This buffer is used to temporarily store the IP packet while its IP address is examined and a proper SP Switch route is set up to transfer the packet through the SP Switch.

Send Controller

and ProcessorThis component is notified when an IP packet is received in the FIFO (2) queue and sets up a DMA transfer to send the packet to Buffer (2). The Send Processor looks up the IP address in the packet header and determines the SP Switch route for the packet, before notifying the Send Controller to send the packet to the Send TBIC from Buffer (2).

Send TBICThis component receives data from Buffer (2) and sends it in SP Switch data segments to the SP Switch.

2.3.7 Media Card Performance

The SP Switch Router adapter has the following performance characteristics:

It is able to transfer up to 100 MB per second. The limiting factor is the crosspoint switch connection bandwidth.

38 IBM 9077 SP Switch Router: Get Connected to the SP Switch

It is able to transfer up to 30,000 packets per second. At 20,000 packets per second, each packet needs to be at 5 KB in order to achieve the 100 MB per second transfer rate mentioned.

As previously mentioned, each adapter stores its own route tables in memory. Therefore, route table lookup is very fast, that is, less than 2.5 µs.

Finally, each media adapter has a 1 Gbit per second dedicated link into the crosspoint switch. That is why the 4-port and 16-port models have an aggregate bandwidth of 4 Gbit and 16 Gbit per second, respectively, for the crosspoint switch.

2.3.8 Other Media Cards

The following are other media cards and adapters currently supported on the GRF:

EthernetThe 10/100 Mb Ethernet media adapter consists of eight 10/100BaseT Ethernet ports. All ports support only shielded twisted pair (STP) and unshielded twisted pair (UTP-5) copper cables. Other types of cables require the user to supply the appropriate transceivers.

ATM OC-3cThe ATM OC-3c media adapter allows the user to connect up to two connections into the ATM network at 155 Mb/s.

ATM OC-12cThe ATM OC-12c media adapter allows the GRF to connect to a single ATM network at speeds of up to 622 Mb per second.

FDDIThe FDDI media card provides four ports in the card. These ports allow the media card to be connected into the Fiber Distributed Data Interchange (FDDI). The four ports can be configured such that they support the following:

Two dual-ring FDDI networks

One dual-ring and two single-ring FDDI networks

Four single-ring FDDI networks

HIPPIThe HIPPI media adapter is a single-port card that allows the GRF to connect to a High Performance Parallel Interface (HIPPI) network at speeds of up to 800 or 1600 Mb/s. After deducting the overhead, this medium can support connections of up to 100 MB/s.

HSSIThe High Speed Serial Interface (HSSI) is a dual-ported media adapter that can connect to two serial networks simultaneously. Each port is capable of up to 45 Mb per second.

Router Node 39

IP/SONETThe IP/SONET OC-3c is a single-ported card that allows the user to connect to a digital network using a transmission format known as Synchronous Optical Network protocol (SONET). This standard is increasingly popular in the telecommunications industry.

2.3.9 GRF Operating Environment

As previously mentioned, the operating temperature should not exceed 53 °C (128 °F). Even though there is a buffer between the operating temperature and the warning temperature, it is best to keep the temperature within the operating level in order to minimize the possibility of damage to GRF components.

2.4 PSSP Enhancements

This section discusses the enhancements made to PSSP to accommodate the Dependent Node Architecture.

2.4.1 SDR Enhancements

As mentioned in Section 2.1.2, “Design Objectives” on page 8, the System Data Repository (SDR) needed to be extended to support the dependent node architecture. Two classes have been added to the SDR.

DependentNode

DependentAdapter

2.4.1.1 DependentNode Attributes

The attributes of the DependentNode class are described in Table 2:

Table 2. DependentNode Attributes

User Defined

System Defined

 

 

node_number

switch_node_number

 

 

extension_node_identifier

switch_number

 

 

reliable_hostname

switch_chip

 

 

management_agent_hostname

switch_chip_port

 

 

snmp_community_name

switch_partition_number

 

 

40 IBM 9077 SP Switch Router: Get Connected to the SP Switch

The attributes of the DependentNode class are described in detail as follows:

AttributeDescription

node_numberThis user-supplied node number represents the node position of an unused SP Switch port used for the SP Switch router adapter.

extension_node_identifierThis is a 2-digit slot number that the SP Switch router adapter occupies on the GRF. Its range is from 00 to 15.

reliable_hostnameThe hostname of the administrative Ethernet, de0, is the GRF’s hostname. Use the long version of the hostname when DNS is used.

management_agent_hostnameThis attribute is the hostname of the SNMP agent for the GRF. For the GRF dependent node, this is the same as the reliable_hostname.

snmp_community_nameThis field contains the SNMP community name that the SP Extension Node SNMP Manager and the GRF’s SNMP Agent will send in the corresponding field of the SNMP messages. This value must match the value specified in the /etc/snmpd.conf file. If left blank, a default name found in the SP Switch Router Adapter documentation is used.

The following attributes are derived by the RS/6000 SP system when the SDR_config routine of endefnode is invoked.

AttributeDescription

switch_node_numberThe switch port that the dependent node is attached to.

switch_numberThe switch board that the dependent node is attached to. switch_chipThe switch chip that the dependent node is attached to.

switch_chip_portThe switch chip port that the dependent node is attached to.

switch_partition_numberThe partition number to which the dependent node belongs.

Router Node 41

2.4.1.2 DependentAdapter Attributes

The attributes of the DependentAdapter class are described in Table 3:

Table 3. DependentAdapter Attributes

User Defined

System Defined

 

 

node_number

-

 

 

netaddr

-

 

 

netmask

-

 

 

The attributes of the DependentAdapter class are described in detail as follows:

AttributeDescription

node_numberThis user-supplied node number represents the node position of an unused SP Switch port to be used by the SP Switch router adapter.

netaddrThis is the IP address of the SP Switch Router adapter. netmaskThis is the netmask of the SP Switch Router adapter.

2.4.1.3 Additional Attributes

As Table 4 shows, additional attributes are added to the Syspar_map_class and the Switch_partition classes:

Table 4. Additional SDR Attributes

Syspar_map class

Switch_partition class

 

 

...

...

 

 

node_type

switch_max_ltu

 

 

 

switch_link_delay

 

 

Details of these attributes follow:

42 IBM 9077 SP Switch Router: Get Connected to the SP Switch

AttributeDescription

node_typeThis attribute is set to dependent for GRF and to standard for all other RS/6000 SP nodes.

switch_max_ltuThis specifies the maximum packet length of data on the SP Switch; the default is 1024. Do not change this value for any reason.

switch_link_delaySpecifies the delay for a message to be sent between the two furthest points on the switch; the default is 31. Do not change this value for any reason.

2.4.2 New Commands

To support the dependent node architecture, seven new commands were added. These commands can be divided into two groups. The first group must only be executed with root permission on the Control Workstation. The second group can be executed by any user on any standard RS/6000 SP node.

Table 5 shows a list of commands of the first group:

Table 5. New Commands (root Executable)

Command

Description

 

 

endefnode

Define or change an dependent node

 

 

enrmnode

Remove a dependent node

 

 

endefadapter

Define or change an dependent node

 

 

enrmadapter

Remove an dependent node adapter

 

 

enadmin

Reconfigure or reset the dependent node

 

 

The first four commands all have the same characteristics, which are as follows:

The are part of the ssp.basic fileset.

They must only be executed on the Control Workstation.

They can only be executed by the root user.

They only affect the current active partition.

They only affect the SDR, unless the -r option is specified (this option is not applicable to enrmadapter).

They return a code of 0 if successful, 1 if failed.

Router Node 43

The enadmin command is used to change the administrative state of a dependent node in the GRF; it has the following characteristics:

It is part of the ssp.spmgr fileset.

It must only be executed on the Control Workstation.

It can only be executed by the root user.

The -r option from endefnode and endefadapter triggers enadmin -a

reconfigure, while the -r option from enrmnode triggers enadmin -a reset.

• The return code is 0 if successful, 1 if failed.

Table 6 shows the list of commands from the second group (executable by any user on any standard node):

Table 6. New Commands (User Executable)

Command

Description

 

 

splstnodes

List SP nodes

 

 

splstadapters

List SP adapter

 

 

These commands have the following characteristics:

They are part of the ssp.basic fileset.

They can be executed on any standard RS/6000 SP node.

They can be executed by any user.

They only affect the current active partition unless the -G option is used.

The following sections describe the commands in more detail.

2.4.2.1 The endefnode Command

The endefnode command can be executed using smit. The fast path for smit is enter_extnode. This command is used to add or change an extension node in the SDR DependentNode class. Its options are shown in Table 7 on page 45.

44 IBM 9077 SP Switch Router: Get Connected to the SP Switch

Table 7. endefnode Command Options

Flags

SMIT Option

Description

 

 

 

-a

Administrative hostname

This is the hostname of the GRF, and the IP name

 

 

of the GRF’s administrative Ethernet, de0. Use

 

 

long names if DNS is used in the network.

 

 

 

-c

SNMP community name

This field contains the SNMP community name

 

 

that the SP extension node SNMP Manager and

 

 

the GRF’s SNMP agent send in the

 

 

corresponding field of the SNMP messages. This

 

 

value must match the value specified in the

 

 

/etc/snmpd.conf file on the GRF. If left blank, a

 

 

default name found in the SP Switch Router

 

 

Adapter documentation is used.

 

 

 

-i

Extension node

This field contains the two-digit slot

 

 

number of the SP Switch Router Adapter

 

 

on the GRF. The value for this field is from

 

 

00-15 and is shown on the slots of the

 

 

GRF.

 

 

 

-s

SNMP Agent hostname

This field refers to the hostname of the

 

 

processor running the SNMP Agent for the

 

 

GRF. In the current version of the GRF, this

 

 

value is equivalent to that of the

 

 

Administrative Hostname.

 

 

 

-r

Reconfigure the

This field specifies whether the enadmin

 

extension node

command is to be activated after the

 

 

endefnode command completes. It is placed

 

 

here so that the user does not have to

 

 

explicitly issue the enadmin command. If the

 

 

specification is yes, the -r option is part of

 

 

the command. If the specification is no, the

 

 

-r option is not part of the command.

 

 

 

 

Node number

This is the node number the extension

 

 

node logically occupies in the RS/6000 SP.

 

 

 

This command adds attribute information for the extension node. The endefadapter command adds IP information, such as the IP address and netmask for the extension node. Together, these two commands define the extension node.

Router Node 45

Attention

Note that this command only affects the SDR, unless the -r option is used. The -r option should be issued only if endefadapter has been executed for the extension node.

When the GRF is properly configured and powered on, with the SP Switch Router Adapter inside, it periodically polls the Control Workstation for configuration data. The -r option or enadmin command is not required to activate the polling here.

2.4.2.2 The enrmnode Command

The enrmnode command is used to remove an extension node from the SDR DependentNode class and can also be executed using smit. The fast path for smit is delete_extnode. Its options are shown in Table 8.

Table 8. enrmnode Command Options

Flags

SMIT Option

Description

 

 

 

-r

Reset the

Specifies whether the enadmin command is to be

 

extension node

activated after the enrmnode command

 

 

completes. With this option the user does not

 

 

have to explicitly issue the enadmin command. If

 

 

the specification is yes, the -r option is part of the

 

 

command. If the specification is no, the -r option

 

 

is not part of the command.

 

 

 

 

Node number

This is the node number the extension node

 

 

logically occupies in the RS/6000 SP.

 

 

 

Attention

Note that this command only affects the SDR, unless the -r option is used. This command should be issued with a -r flag, because the enadmin command is not available for the extension node after enrmnode is executed, since the extension node has been removed from the SDR.

46 IBM 9077 SP Switch Router: Get Connected to the SP Switch

2.4.2.3 The endefadapter Command

The endefadapter command is used to add or change the extension node adapter IP information in the SDR DependentAdapter object, and can be executed using smit. The fast path for smit is enter_extadapter. The command options are shown in Table 9.

Table 9. endefadapter Command Options

Flags

SMIT Option

Description

 

 

 

-a

Network

Specifies the IP address of the extension node.

 

address

 

 

 

 

-m

Network

Specifies the netmask for the extension node.

 

netmask

 

 

 

 

-r

Reconfigure

Specifies if the enadmin command is to be

 

the extension

activated after the endefadapter command

 

node

completes. With this option, the user does not

 

 

have to explicitly issue the enadmin command. If

 

 

the specification is yes, the -r option is part of the

 

 

command. If the specification is no, the -r option

 

 

is not part of the command.

 

 

 

 

Node number

This is the node number the extension node

 

 

logically occupies in the RS/6000 SP.

 

 

 

Attention

Note that this command only affects the SDR, unless the -r option is issued. The -r option should be issued only if the endefnode has been executed for the extension node.

When the GRF is properly configured and powered on, with the SP Switch Router Adapter inside, it periodically polls the Control Workstation for configuration data. The -r option or enadmin command is not required to activate the polling here.

2.4.2.4 The enrmadapter Command

The enrmadapter command is used to remove the SDR DependentAdapter object, and can also be executed using smit. The fast path for smit is delete_extadapter.

Router Node 47

2.4.2.5 The enadmin Command

The enadmin command is used to change the status of the SP Switch router adapter in the GRF and can also be executed using smit. The fast path for smit is manage_extnode. The command options are shown in Table 10.

Table 10. enadmin Command Options

Flags

SMIT Option

 

Description

 

 

 

 

-a

Actions to

be

Either reset or reconfigure. A reset is sent to the

 

performed

on

extension node SNMP Agent to change the

 

the extension

target node to a down state (not active on the SP

 

node

 

Switch). A reconfigure is sent to the extension

 

 

node SNMP Agent to trigger reconfiguration of

 

 

 

 

 

 

the target node, which causes the SNMP Agent

 

 

 

to request new configuration parameters from

 

 

 

the SP extension node SNMP Manager, and to

 

 

 

reconfigure the target node when the new

 

 

 

parameters are received.

 

 

 

 

 

Node number

 

This is the node number the extension node

 

 

 

logically occupies in the RS/6000 SP.

 

 

 

 

48 IBM 9077 SP Switch Router: Get Connected to the SP Switch

2.4.2.6 The splstnodes Command

The splstnodes command is used to list the node attributes of all nodes in the SDR, and can also be executed using smit. The fast path for smit is list_extnode. See all command options in Table 11.

Table 11. splstnode Command Options

Flags

Description

 

 

-h

Outputs usage information.

 

 

-G

Ignores partition boundaries for that output.

 

 

-x

Inhibits header record in the output.

 

 

-d <delimiter>

Uses the <delimiter> between its attributes in the output.

 

 

-p <string>

Uses the <string> value in the output in place of an

 

attribute that has no value.

 

 

-s <attr>

Sorts the output using the <attr> value. In SMIT, this field

 

is known as Sort Attribute.

 

 

-t <node_type>

Uses standard to list RS/6000 SP nodes, or dependent. If

 

none is specified, it displays standard and dependent. In

 

SMIT, this field is known as Node Type.

 

 

-N <node_grp>

Restricts the query to the nodes belonging to the node

 

group specified in <node_grp>. If the <node_grp>

 

specified is a system node group, the -G flag is implied.

 

 

<attr==value>

This operand is used to filter the output, such that only

 

nodes with attributes that are equivalent to the value

 

specified are displayed. In SMIT, this field is known as

 

Query Attribute.

 

 

<attr>

This is a list containing attributes that are displayed by the

 

command. If none is specified, it defaults to node number.

 

This list of attributes can be found in the DependentNode

 

class. In SMIT, this field is known as Attribute.

 

 

Router Node 49

2.4.2.7 The splstadapters Command

The splstadapers command is used to list the adapter attributes of all nodes in the SDR, and can also be executed using smit. The fast path for smit is list_extadapter. See all command options in Table 12.

Table 12. splstadapter Command Options

Flags

Description

 

 

-h

Outputs usage information.

 

 

-G

Ignores partition boundaries for its output.

 

 

-x

Inhibits header record in the output.

 

 

-d <delimiter>

Uses the <delimiter> between its attributes in the output.

 

 

-p <string>

Uses the <string> value in the output in place of an

 

attribute that has no value.

 

 

-t <node_type>

Uses standard to list RS/6000 SP nodes, or dependent. If

 

none is specified, it displays standard and dependent. In

 

SMIT, this field is known as Node Type.

 

 

<attr==value>

This operand is used to filter the output, such that only

 

nodes with attributes that are equivalent to the value

 

specified are displayed. In SMIT, this field is known as

 

Query Attribute.

 

 

<attr>

This is a list containing attributes that are displayed by the

 

command. If none is specified, it defaults to node number.

 

This list of attributes can be found in the Adapter and

 

DependentAdapter class. In SMIT, this field is known as

 

Output Attribute.

 

 

50 IBM 9077 SP Switch Router: Get Connected to the SP Switch

2.4.3 Enhanced Commands

The following commands (see Table 13) have been modified due to the introduction of the dependent node:

Table 13. Enhanced Commands

Command

Comment

 

 

Eprimary

The dependent node cannot be the Primary node.

 

 

Estart

The dependent Node depends on the Primary node to calculate the

 

routes.

 

 

Efence

Enhanced for dependent node support.

 

 

Eunfence

Enhanced for dependent node support.

 

 

Here is a more detailed description about the modifications:

•Eprimary

This command has been modified so that dependent nodes will not be able to act as a Primary or Primary Backup node for the SP Switch in the partition. The dependent node does not run the RS/6000 SP Switch codes like standard RS/6000 SP nodes and therefore does not have the ability to act as the Primary or Primary Backup node.

•Estart

This command functions as it does with normal nodes. It was enhanced to support the depend node in the RS/6000 SP.

•Efence

This command functions as it does with normal nodes in the RS/6000 SP. In addition, the dependent node can be fenced from the SP Switch with autojoin like any other standard RS/6000 SP node.

•Eunfence

This command functions as it does with normal nodes in the RS/6000 SP. In addition, the dependent node can rejoin the SP Switch network with this command, if that node was previously removed from the switch network due to failures or Efence.

Router Node 51

2.4.4 Hardware Perspectives

In Perspectives IP Node is used as a convenient and short descriptive term easily displayed in the GUI. It conveys the role and functions of the dependent node. Currently, this is the only dependent node.

In Figure 20 we show the changes made to Perspectives because of the introduction of the IP Node. The changes are restricted to the Hardware and System Partition Aid Perspectives.

1

2

3

4

Figure 20. Hardware Perspectives

This figure shows the Hardware Perspectives, which can be started using the command perspectives and selecting the Hardware icon. Alternatively, it can be started directly via the command sphardware.

The Hardware Perspective consists of the following four parts:

1.Menu bar

2.Toolbar

52 IBM 9077 SP Switch Router: Get Connected to the SP Switch

3.Nodes pane (Frame or Icon View)

4.Information area

The most obvious change is the addition of the IP Node icon as seen in the Nodes pane. (The figure above shows the Frame View.) The default label for this icon is IP Node <node number>.

The IP Node icon is also located on the side of the frame, where a standard node with that node number would be. In this figure the IP Nodes are 7, 14 and 15.

When switch_responds is monitored, it shows the IP Node in two states:

Green when working with the SP Switch.

Marked with a red cross when fenced or not operating due to hardware or configuration problems.

In the figure, IP Node 7 and 15 are working, while IP Node 14 is down.

2.4.4.1 Action Menu

In Figure 21 on page 54, we see that IP Node 7 is selected in the Nodes pane, and Actions->Nodes is selected in the menu bar (1). We see that only the following five actions are available:

Router Node 53

1

2

3

4

Figure 21. Action Menu

View

This will bring up the IP Node’s hardware notebook, shown in the next figure.

Fence/Unfence...

This will bring up another window to allow us to either fence or unfence an IP Node. If we are fencing the IP Node, we can use the option of autojoin.

Create Node Group...

This will bring up another window to allow us to add the RS/6000 SP nodes to a Node Group. This action does not affect the IP Node, even though it is selectable.

3 Digit Display

54 IBM 9077 SP Switch Router: Get Connected to the SP Switch

This will bring up a window to show the three-digit display of all RS/6000 SP standard nodes in the current partition. This action does not apply to the IP Node, even though it is selectable.

Open Administrative Session...

This action will open a window that is a Telnet session to the GRF, using the reliable_hostname attribute specified in the DependentNode class.

In addition, the Nodes pane in this figure shows the Icon View. In this view, the IP Node icons are always located after all the standard RS/6000 SP node icons. The results of monitoring the IP Nodes and the icon labels are the same as those of Frame View, mentioned in the previous figure.

2.4.4.2 Hardware Notebook

Figure 22 shows the IP Node hardware notebook. This notebook can be triggered by selecting the Notebook icon on the Hardware Perspective toolbar (2), or selecting Action->Nodes->View in the menu bar (1).

Figure 22. Hardware Notebook

The notebook has three tabs: Configuration, All Dynamic Resource Variables, and Monitored Conditions. This figure shows the Configuration tab.

Router Node 55

These are the attributes listed in the Configuration tab:

Node number

Hostname

Management agent hostname

SNMP community name

System partition

Extension node identifier

Dependent node IP address

Dependent node netmask

Switch port number

Switch number

Switch chip

Switch chip port

Switch partition number

Switch responds

The All Dynamic Resource Variables tab only shows the state of the Switch Responds, and the Monitored Conditions tab only shows the value of the Switch Responds if it is being monitored.

2.4.4.3 System Partition Aid Perspectives

The System Partition Aid Perspectives window in Figure 23 on page 57 has two panes, the Nodes pane and the System partitions pane. The Nodes pane

(3) in this figure shows the Icon view. Notice that the IP Nodes are displayed after all the standard RS/6000 SP nodes.

56 IBM 9077 SP Switch Router: Get Connected to the SP Switch

1

2

3

4

Figure 23. System Partition Aid Perspectives

The IP Nodes can only be assigned to a partition here. This is done either by using the Assign icon in the toolbar (2), or by selecting

Action->Nodes->Assign Nodes to System Partition on the menu bar (1). Except for the System Partition Notebook, discussed in the next figure, all other actions, though selectable, do not apply to the IP Node.

2.4.4.4 System Partition Aid Notebook

Figure 24 on page 58 shows the IP Node System Partition Aid Notebook. This notebook can be triggered by selecting the Notebook icon on the Hardware Perspective toolbar (2), or selecting Action->Nodes->View on the menu bar

(1).

The notebook only has the Node Information tab shown in this figure.

Router Node 57

Figure 24. System Partition Aid Notebook

These attributes are listed in the Node Information tab:

Node number

Switch port number

Assigned to system partition

2.4.5 SP Extension Node SNMP Manager

The SP Extension Node SNMP manager is contained in the ssp.spmgr file set of PSSP. This file set must be installed on the Control Workstation in order for the GRF to function as an extension node.

The SP Extension Node SNMP manager is an SNMP manager administered by the System Resource Controller. The purpose of the SNMP manager is to communicate with the SNMP agent on the GRF. The SNMP manager and the agent adhere to Version 1 of the SNMP protocol. The SNMP manager sends configuration data for an extension node to the SNMP agent on the GRF. The SNMP agent applies the configuration data to the SP Switch Router Adapter represented by the extension node. The SNMP agent also sends asynchronous notifications in the form of SNMP traps to the SNMP Manager when the extension node changes state. The following commands are available to control the SP Extension Node SNMP Manager:

•startsrc

58 IBM 9077 SP Switch Router: Get Connected to the SP Switch

•stopsrc

•lssrc

•traceson

•tracesoff

2.4.6 Dependent Node MIB

IBM has defined a dependent node SNMP Management Information Base (MIB) called ibmSPDepNode. This MIB contains definitions of objects representing configuration attributes of each dependent node and its state. The GRF Agent maintains the state and configuration data for each dependent node using the MIB as a conceptual database.

The MIB defines a single table of up to 16 entries representing the adapter slots in the GRF. When a slot is populated by an SP Switch Router Adapter, the entry in the table, accessed using the extension node identifier, contains the configuration attribute and state values for the adapter in the slot. Also included in the MIB are the definitions of trap messages sent by the GRF agent to the SP Extension Node SNMP Manager. A copy of the MIB is contained in the file /usr/lpp/ssp/config/spmgrd/ibmSPDepNode.my on the Control Workstation.

Other SNMP managers in the network can query this MIB table to validate the configuration and status of the dependent node and GRF. However, only an SNMP manager using the correct SNMP community name can change the values in the MIB table.

Following is a listing of MIB entries:

EntryDefinition

ibmSPDepNodeObject identifier for the dependent node in the MIB database.

ibmSPDepNodeTableTable of entries for dependent nodes.

ibmSPDepNodeEntryA list of objects comprising a row and a clause in ibmSPDepNodeTable. The clause indicates which object is used as an index into the table to obtain a table entry.

ibmSPDepNodeNameThe extension_node_identifier attribute in the DependentNode class.

ibmSPDepNodeNumberThe node_number attribute in the DependentNode class.

Router Node 59

ibmSPDepSwTokenA combination of switch_number, switch_chip and switch_chip_port attributes from the DependentNode class.

ibmSPDepSwArpThe arp_enabled attribute in the Switch_partition class.

ibmSPDepSwNodeNumberThe switch_node_number attribute in the DependentNode class.

ibmSPDepIPaddrThe netaddr attribute in the DependentAdapter class. ibmSPDepNetMaskThe netmask attribute in the DependentAdapter class.

ibmSPDepIPMaxLinkPktThe switch_max_ltu attribute in the Switch_partition class.

ibmSPDepIPHostOffsetThis attribute stores the difference between the host portion of a node’s IP address and its corresponding switch node number. When ARP is disabled on the SP Switch network, this offset is subtracted from the host portion of the IP address to calculate the switch node number.

ibmSPDepConfigStateThe six config states of the dependent node are: notConfigured, firmwareLoadFailed, driverLoadFailed, diagnosticFailed, microcodeLoadFailed, and fullyConfigured, for use in configuring the adapter.

ibmSPDepSysNameThe syspar_name attribute in the Syspar class.

ibmSPDepNodeStateThe value of nodeUp or nodeDown, to show the status of the dependent node.

ibmSPDepSwChipLinkThe switch_chip_port attribute in the DependentNode class.

ibmSPDepNodeDelayThe switch_link_delay attribute in the Switch_partition class.

ibmSPDepAdminStateThe value of up, down, or reconfigure, indicating the desired state of the dependent node. If the dependent node is not in its desired state, the SNMP agent on the GRF will trigger the appropriate action to change its state.

2.4.7 Coexistence

Figure 25 shows a single-frame RS/6000 SP in a single partition with a connection to the GRF. Nodes 1 and 2 are installed with PSSP 2.4. The other nodes are installed with any other version of PSSP that can coexist with

60 IBM 9077 SP Switch Router: Get Connected to the SP Switch

PSSP 2.4 to represent coexistence. Also, note that Node 16 is empty, because the SP Switch port for this node is used by the SP Switch router adapter in the GRF.

 

 

 

PSSP 2.4 or PSSP2.3 and IX70649 on

 

 

PSSP 16

PSSP

CWS

 

 

 

15

Primary switch node

 

 

 

*.*

*.*

 

 

13

PSSP 14

PSSP

Backup switch node

 

 

 

*.*

*.*

PTFs for all other nodes

 

 

11

PSSP 12

PSSP

 

 

 

*.*

*.*

for PSSP2.1 IX71246

 

 

9

PSSP 10

PSSP

 

 

for PSSP2.2 IX71245

 

 

 

*.*

*.*

 

 

7

PSSP

8 PSSP

ssp.spmgr file set installed on CWS

 

 

*.*

*.*

 

5

PSSP

6 PSSP

Must be an SPS or SPS-8 switch

 

 

*.*

*.*

 

 

 

 

3

PSSP

4 PSSP

 

 

 

 

 

*.*

*.*

 

 

IP Switch

 

1

PSSP

2 PSSP

 

 

 

 

 

Control

 

 

2.4

2.4

SP Switch Router Cable

 

 

Board

Crosspoint

 

Switch

 

 

 

 

 

SP Switch

Switch

 

 

 

 

 

 

Frame

Ethernet

 

4-port FDDI

 

 

 

 

Cable

CWS

GRF 400

 

 

 

 

 

 

 

 

RS232

 

 

 

 

 

 

Cable

 

 

 

 

 

 

 

PSSP 2.4

 

 

Figure 25. Coexistence

The dependent node is only supported in PSSP 2.3 and higher PSSP versions. To use it with nodes with PSSP versions less than 2.3 requires the use of coexistence. The following conditions are required for the dependent node to communicate with nodes with a lower version than 2.3 using coexistence:

The Control Workstation must be at PSSP 2.3 or higher to manage dependent nodes.

The Primary node of the SP Switch must be at PSSP 2.3 or higher, as the Primary node needs to perform some tasks for the dependent node and these functions are only available in PSSP 2.3 and higher PSSP versions.

The Primary Backup node of the SP Switch should be PSSP 2.3 or higher so that if the Primary node fails, the dependent node can continue to function in the RS/6000 SP when the Backup node takes over.

Router Node 61

All RS/6000 SP nodes with a version less than PSSP 2.3 in the partition need to maintain the right level of fixes (PTFs) in order for coexistence with PSSP 2.4 to take place.

The ssp.spmgr file set must be installed on the Control Workstation.

Because the SP Switch router adapter will only work with the 8-port or 16-port SP Switch, make sure that the switch used in the RS/6000 SP is not a High Performance Switch (HiPS).

There must be at least one free SP Switch port to install the SP Switch router adapter.

Important

When the Primary switch node fails, the Primary Backup Switch node take over as the new Primary switch node. The new Primary Backup switch node, selected from the current partition, can be a node with a PSSP level below 2.3, even though another node with a PSSP level of 2.3 or higher may exist in that partition. The only way to ensure that the new Backup switch node is at PSSP 2.3 or higher is to manually check the RS/6000 SP system. If the PSSP Version of the Primary Backup switch node is below Version 2.3 you have to chose a node with PSSP 2.3 or higher as the Primary Backup switch node.

If a node running a version of PSSP earlier then 2.3 is selected as the new primary node, the SP Switch Router Adapter will be fenced from the switch.

2.4.8 Partitioning

Figure 26 on page 63 shows a single-frame RS/6000 SP broken into two partitions, Partition A and Partition B. Each partition has seven standard RS/6000 SP nodes and one dependent node. Only seven nodes are allowed in each partition, as a single-frame RS/6000 SP has only 16 SP Switch ports, and two of them are used for the SP Switch router adapter, one for each partition.

62 IBM 9077 SP Switch Router: Get Connected to the SP Switch

 

 

Cross-partition communication

 

 

through the SP Switch

 

 

IP Switch

 

 

 

Control

 

 

 

Board

Crosspoint

 

 

SP Switch

Switch

Partition B

 

SP Switch

 

 

4-port FDDI

 

 

 

 

 

 

GRF 1600/400

Partition A

 

Partition A

Partition B

 

SDR

Switch

 

 

 

Frame

Ethernet

 

 

 

Cable

CWS

 

 

RS232

 

 

 

Cable

 

 

 

 

PSSP 2.4

 

Figure 26. Partitioning

Normally, RS/6000 SP nodes in different partitions cannot communicate with each other through the SP Switch. The GRF plays a unique role here by allowing RS/6000 SP nodes to communicate across partitions, when each partition contains at least one SP Switch router adapter, and these adapters are interconnected by TCP/IP.

The requirements for partitioning are the same as those for coexistence, with the addition of having at least one free SP Switch port per partition, to connect to the SP Switch router adapter.

2.5 Planning for the GRF

Before acquiring any model of the SP Switch Router, ensure that there are SP Switch ports available in the designated partition, and that the switch used in the RS/6000 SP is the 8-port or 16-port SP Switch.

Router Node 63

Next, ensure that the following parameters are defined:

ParametersDescriptions

GRF IP addressThe IP address for the GRF administrative Ethernet.

GRF netmaskThe Netmask for the GRF administrative Ethernet.

GRF Default routeThe default route of the GRF.

SNMP community nameThis attribute describes the SNMP community name that the SP Extension Node SNMP Manager and the GRF’s SNMP Agent will send in the corresponding field of the SNMP messages. This value must match the value specified for the same attribute of the corresponding dependent node definition on the SP system. If left blank, a default name found in the SP Switch Router Adapter documentation is used.

CWS IP addressThe Control Workstation’s IP address. When a GRF contains multiple SP Switch router adapters that are managed by different SNMP managers on different RS/6000 SP CWS, each of the Control Workstation IP addresses should be defined along with a different community name for each Control Workstation.

DNSThe DNS server and domain name, if used.

SP Extension Node SNMP

Manager port #The SNMP port number used by the SP Extension Node SNMP Manager to communicate with the SNMP agent on the GRF.

This port number is 162 when the SP Extension Node SNMP Manager is the only SNMP manager on the Control Workstation. Otherwise, another port number not used in the /etc/services of the Control Workstation is chosen.

64 IBM 9077 SP Switch Router: Get Connected to the SP Switch

2.6 Planning for the Dependent Node

Next, for each dependent node on the RS/6000 SP, define the following:

ParametersDescriptions

Node #A user-supplied dependent node number representing the node position of an unused SP Switch port to be used by the SP Switch Router Adapter.

Slot #The slot number on which the SP Switch Router Adapter is located in the GRF.

GRF hostnameThe hostname for the GRF administrative Ethernet. A long hostname is recommended if the domain name service (DNS) is used in the network. This represents both the Administrative and SNMP agent hostname of the dependent node.

SNMP community nameThis attribute describes the SNMP community name that the SP Extension Node SNMP Manager and the GRF’s SNMP Agent will send in the corresponding field of the SNMP messages. This value must match the value specified in the /etc/snmpd.conf file on the GRF. If left blank, a default name found in the SP Switch Router Adapter documentation is used.

SP Extension Node SNMP

Manager port #The SNMP port number used by the SP Extension Node SNMP Manager to communicate with the SNMP agent on the GRF.

This port number is 162 when the SP Extension Node SNMP Manager is the only SNMP manager on the Control Workstation. Otherwise, another port number not used in the /etc/services of the Control Workstation is chosen.

Then, for the dependent node adapter, define these parameters:

ParameterDescriptions

IP addressThe IP address of this adapter.

NetmaskThe netmask of this adapter. Use the same format as that for standard RS/6000 SP nodes.

Router Node 65

2.7 Conclusion

The SP Switch Router 9077-04S has an aggregate bandwidth of 800 MB/s. An SP wide node by contrast is capable of no more than about 65 MB/s of sustained throughput. A wide node’s CPU hits a wall at about 5000 packets/second, whereas the 9077 is capable of an aggregate of 2.8 million packets/second. All this is achieved in part because of the non-blocking crosspoint switch with four 100 MB/s, full duplex connection points. This enables multiple paths to operate at full speed simultaneously.

Unlike the SP nodes, the SP Switch Router is designed with high availability in mind. It provides balanced, fully redundant power supplies that can be hot swapped in case of failure. It provides the ability for redundant paths to an SP Switch to be configured on a single 9077; with dynamic routing protocols, a second 9077 can be used to provide a backup path in case of system failure of the primary router. In either case, each media card is hot swappable and autoconfigured after the initial install has been completed.

With its high port count on interfaces such as FDDI and Ethernet and its highly scalable performance, the SP Switch Router provides a very cost effective solution. With each media card you get a nearly linear scaling of performance with very little cost increase. An SP node by comparison runs out of CPU cycles and/or slots very quickly requiring the purchase of another entire node.

Since the 9077 (or rather the Ascend GRF) was originally designed for ISP’s, it has a full set of protocols, including dynamic routing protocols such as OSPF, BGP4 and RIPv2. It also has the memory required to hold up to 150,000 routes and the speed to access a table of this size without performance degradation. Support for media types not supported by the SP nodes also enables the SP to now be connected into networks that will be important for its future. These include support for HSSI and Sonet, which are important for the SP’s ever-growing role as a Web server or online transaction manager.

66 IBM 9077 SP Switch Router: Get Connected to the SP Switch

Part 2. Scenarios

This part presents some sample configurations of an RS/6000 SP system with an SP Switch Router. It is beyond the scope of this book to represent all possible applications of an SP Switch Router. Nevertheless, the basic configurations shown are building blocks for more complex networking topologies that include the SP Switch Router and may inspire more complex configurations.

All following sample scenarios were carefully chosen to match frequently occurring customer situations. They can be easily configured and modified to apply to customers’ needs. All configurations described were tested in our laboratory with the available hardware and software. We used two RS/6000 SPs, each consisting of a control workstation (CWS) and one frame with an SP Switch and several nodes (see Figure 27). All nodes and the CWS were installed with AIX 4.3.1 and PSSP 2.4. Additionally, a GRF1600 and a GRF400 running Ascend Embedded/OS V1.4.6.4. were used. Only static routing was applied in our tests. For using and configuring gated on an SP Switch Router refer to GRF Reference Guide 1.4, GA22-7367. For detailed configuration information, refer to Appendix A, “Laboratory Hardware and Software Configuration” on page 233, and the scenario sections.

SP 21

SP 2

 

GRF 400

 

GRF 1600

CWS 21

CWS 2

Figure 27. The Laboratory Hardware Installation

But first let us get physical and see how the SP Switch Router media cards are configured.

© Copyright IBM Corp. 1998

67

68 IBM 9077 SP Switch Router: Get Connected to the SP Switch

Chapter 3. Installation and Configuration

The SP Switch Router functions as an IP router to provide high-speed data communication links between SP processor nodes and external networks or hosts. The SP Switch Router Adapter media card connects to the SP Switch board in an SP system as shown in Figure 28.

Administrative network =

SP Control Workstation

 

 

 

 

 

Ethernet hub or bridge

 

 

SP Switch Router

Control

 

 

 

 

board

 

 

 

Switch

 

Processor

Processor

Primary node

 

SP Switch

node

node

for SP Switch

 

 

 

 

 

Router Adapter

 

 

 

 

media card

 

 

 

 

 

 

SP Switch

 

to/from other networks and hosts

 

 

 

Figure 28. Connecting the GRF to the SP Switch and the CWS

The SP Switch Router Adapter card also transmits data to/from other types of media cards across the SP Switch Router’s internal switch core. These media cards include HIPPI, HSSI, FDDI, ATM OC-3c, ATM OC-12c, 100Base-T (Fast Ethernet), and other SP Switch Router Adapter cards. The SP system manages the SP Switch Router Adapter card as a dependent node, under the control of the SP SNMP Manager running on the SP Control Workstation and the Primary node of the SP Switch. To learn more about dependent nodes, see Chapter 1, “Dependent Node” on page 3. Once powered on and started up, the SP Switch Router can be configured and managed remotely, either via a site’s administrative network, or using Telnet from the CWS.

Information about procedures performed from the SP CWS are found in the "Managing Extension Nodes" chapter in RS/6000 SP: Administration Guide Version 2 Release 4, GC 23-3897.

© Copyright IBM Corp. 1998

69

The intent of this chapter is to provide, or refer you to, the necessary information to enable you to attach an SP Switch Router to an IBM SP system. Coverage is provided as follows:

Information to configure the SP Switch Router Adapter card as required for SP Switch Router functionality is complete in this chapter.

Information to physically connect the two independent systems across cables is complete in this chapter.

Information to start up, configure, and begin operations on the SP Switch Router is contained in GRF 400/1600 Getting Started 1.4, GA22-7368.

Information to configure the SP Switch Router Adapter card as required for SP system functionality is only partially described in this chapter. Detailed information is contained in the "Managing Extension Nodes" chapter in RS/6000 SP: Administration Guide Version 2 Release 4, GC233897.

3.1 Initial Configuration

When a new, unconfigured GRF is powered on an initial configuration script runs automatically. You will be asked a series of questions about the configuration, as shown in the following screen.

Welcome to Ascend Embedded/OS system configuration..

Host name for this machine ?

Do you wish to configure the maintenance Ethernet interface ?

Which interface type ? (TP BNC AUI)

IP address of this machine ?

Netmask for this network ?

IP address of router (‘none’ for no default route)?

Do you wish to go through the questions again ?

Which region is this machine located in ?

Which time zone ?

Next you are prompted to change the local password for root.Changing the root password is the end of the configuration script. If you need to change these parameters later, run the config_netstat script again. More details about the initial configuration are in GRF Configuration Guide 1.4, GA22-7366.

If you log onto the GRF, a super> prompt appears. This indicates that you are in the command-line interface (CLI). This CLI is different compared to other

Installation and Configuration

70

UNIX systems. On most of the UNIX systems you are working on the shell layer after you logged onto the system.

Many system management and configuration commands are now available. Enter a question mark (?) to retrieve a list of CLI commands. To edit configuration files, you must be in the UNIX shell. The sh command opens the UNIX shell you use to modify configurations. The following screen shows how to do this.

August 22 14:18:31 grf16 kernel: ge027: GRF Ethernet, GRIT address 0:2:7 super>

super>ls

super>no profile was specified super>sh

Copyright 1992, 1993, 1994, 1995, 1996 Berkeley Software Design, Inc. Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994

The Regents of the University of California. All rights resevered.

Ascend Embedded/OS GR TA1.4.6 Kernel #1 (nit): Fri Jan 30 13:08:03 CST 1998

Ascend Embedded/OS 1.4.6

Copyright 1992,1993,1994,1995,1996,1997,1998 Ascend Communications, Inc.

IMPORTANT: By use this software you become subject to the terms and conditions of the license agreement on file /etc/license and any other License agreements previously provided to you bt Ascend Communications. #ls

.chsrc .klogin .login .profile

#

We found it more convenient to work directly at the shell layer. Therefore, we modified the .profile and commented the appropriate part that starts the CLI. For more details, see Appendix B.1, “/root/.profile” on page 261.

3.2 Pre-Installation Assumptions

We assume the following:

The RS/6000 SP Switch Router is powered on and has a VT-100 or administrative Ethernet network connected to its control board.

Figure 29 on page 73 and Figure 30 on page 74 illustrate correct connections and proper setup.

The SP Switch Router’s basic system parameters, such as the IP address and host name, were configured during the first power-on configuration script. You use the terminal or network to log in to the SP Switch Router

Installation and Configuration

71

system and enter these basic configuration parameters. Procedures for starting and setting up the SP Switch Router are found in GRF 400/1600 Getting Started V.14, GA22-7368.

Ignore the prompts for network logging, since we will configure logging to a PCMCIA device; just press Enter when asked to enter the remote logging host name or its IP address.

Remote Telnet access is working. To enable it, you have to edit the /etc/ ttys file on the SP Switch Router and modify the appropriate entries, as follows:

#name

getty

type

status

comments

ttyp0

none

network secure

 

ttyp1

none

network secure

 

ttyp2

none

network

 

 

ttyp3

none

network

 

 

Adding secure to a ttypX stanza opens it for Telnet, so in this example, only two Telnet sessions at a time were allowed.

Use the following command to connect to the SP Switch Router:

xterm -sb -geometry 80x65 -e telnet <hostname of SP Switch Router>

This gives you a large window with a scroll bar at the left, so that you may retrieve information easier. See Appendix B.1, “/root/.profile” on page 261 for a modified .profile for the root user.

Hint: If you just use -e telnet and give the hostname at the telnet> prompt, the window will survive reboots of the GRF.

Installation and Configuration

72

 

 

Standard Switch Cable of 10m

 

 

Other Switch Cables

 

 

 

5 m

(f/c 9305)

 

 

 

 

10 m

(f/c 9310)

 

 

Ethernet

CWS

15 m

(f/c 9315)

 

 

Cable

 

 

 

RS232

 

20 m

(f/c 9320)

 

 

Cable

PSSP 2.4

 

 

 

 

 

 

 

 

 

 

 

 

 

IP Switch

 

 

 

10BaseT

Control

 

 

 

 

 

Board

Crosspoint

 

 

 

 

SP Switch

 

 

SP Switch Cable

Switch

 

 

 

 

 

Grounding Cable

 

 

Figure 29. Connecting the GRF to the Frame

You are ready to configure media cards. Procedures to configure media cards are in this redbook; complete information is in the GRF Configuration Guide 1.4, GA22-7367.

Installation and Configuration

73

 

 

 

Admin Ethernet

 

cws

 

IP Switch

(de0)

 

 

 

Terminal Settings

 

Control

 

Crosspoint

Board

 

9600 baud

 

SP Switch

 

 

Switch

 

 

 

RS232

 

 

 

No parity

 

 

 

GRF 400

 

 

(Null

Eight data bits

 

 

 

Modem

 

 

 

Cable)

One stop bit

VT100 terminal

GRF Console (optional)

Figure 30. Connecting the GRF Console

The IBM SP system is up and operating.

The SP system administrator has given you one of these pieces of information:

The node number assigned to each SP Switch Router Adapter card to be attached to an SP Switch port

The port location on each SP Switch reserved for specific SP Switch Router Adapter cards

3.2.1 Order of Information

Here is the sequence of steps you have to complete:

1.An installation overview of tasks involving the SP Switch Router, the SP Switch Router Adapter card, and the SP system

2.The configuration procedure for the PCMCIA 520 MB disk, which also initiates system logging

3.A description of which cables to attach between the SP Switch Router and the SP control workstation, and between the SP Switch Router Adapter card and the SP Switch

Installation and Configuration

74

4.Methods to determine node number and SP Switch port for an SP Switch Router Adapter card

5.A step-by-step configuration of an SP Switch Router Adapter card

6.A list of ways to verify that the SP Switch Router Adapter card is correctly installed in the SP Switch Router

7.A description of what needs to occur to bring the card online to the SP system

3.3 Installing an SP Switch Router Adapter Card

This section contains the procedure for physical installation and minimal configuration of the SP Switch Router Adapter card for use as an SP dependent node. This includes cabling the GRF to the SP CWS and the appropriate SP Switch port.

Note: There needs to be a network path between Ethernet twisted-pair interface on the SP Switch Router control board and the SP control workstation. This is most easily done through an Ethernet hub (or bridge to the 10BaseT SP LAN). However, it can also be done through a connection to a network external to the SP.

3.3.1 Installation Overview

IBM support personnel who install the SP Switch Router (9077) perform the physical installation and minimal configuration described below with help from the customer’s system administrator. The system administrator must complete the following basic configuration steps:

1.Locate all the components of the SP Switch Router chip group.

2.Perform the complete physical installation of the SP Switch Router unit as described in the "Power On and Initial Configuration" chapter of GRF 400/ 1600 Getting Started 1.4, GA22-7368. Make sure that when the "First-time power on configuration script" runs at system boot, the required configuration information is provided by or entered by the customer. This information includes the SP Switch Router unit IP address and hostname. As stated before, ignore script references to network or syslog logging.

3.Perform the procedure to configure the PCMCIA disk. The procedure is included in Section 3.3.2, “Installing the PCMCIA Spinning Disk” on page 76.

Installation and Configuration

75

4.Route the Ethernet twisted-pair cable between the SP Switch Router unit and the Ethernet hub, then connect the cable to the SP Switch Router control board and the Ethernet hub.

5.Verify that the SP CWS has a connection to this same Ethernet hub. If the SP CWS Ethernet adapter is configured by the system administrator, then a ping test from the SP CWS to the configured SP Switch Router Ethernet address should be done to test Ethernet connectivity.

Physical installation and minimal configuration are complete at this point.

Review Section 3.4, “Attaching SP Switch Router Cables” on page 79 before connecting the SP Switch Router Adapter card cables to the SP Switch ports specified for this configuration.

3.3.2 Installing the PCMCIA Spinning Disk

Your system is shipped with a PCMCIA disk that is required to collect the system log files. This disk can hold up to 520 MB of data regarding your model of the SP Switch Router.

You can install the disk any time after the SP Switch Router is powered on and running. Logging is not enabled until you install the disk and complete this configuration procedure. Logged messages might be very helpful while you are configuring media cards. The configuration is done only once to set up local logs and dumps, and is not affected by software updates or system reboots. System logs include: gritd.packets, grinchd.log, gr.console, gr.conferrs, gr.boot and mib2d.log.

The procedure formats and initializes an external flash (/dev/wdXa), where the X is normally a 1, denoting the number of the device. You get the actual number from the mountf command. We have /dev/wd3a in our example and this value will be used for the rest of this chapter. The procedure then mounts the flash temporarily on /mnt and creates subdirectories, symbolic links and a permanent site file for storing the symbolic links.

Proceed as follows:

1.Insert the PCMCIA disk into slot A on the SP Switch Router control board (the width of the disk requires it to be installed in slot A).

2.Log in as root to the SP Switch Router, start the UNIX shell, and execute the following commands from the shell:

Installation and Configuration

76

prompt> sh

#

#cd /

#iflash -A

May 29 15:54:18 grf16 kernel: wd2: no disk label

# mountf -A -w -m /mnt

Device /dev/wd3a mounted on /mnt

#mkdir /mnt/crash

#mkdir /mnt/portcards

#cd /var

#mv crash crash.orig

#mv portcards portcards.orig

#ln -s /var/log/portcards /var/portcards

#ln -s /var/log/crash /var/crash

#grsite --perm portcards crash

Device /dev/wd0a mounted on /flash. Device /dev/wd0a unmounted.

#

#cd /var/log

#pax -rw -pe -v . /mnt /mnt/.

/mnt/./cron

/mnt/./daemon.log

/mnt/./lastlog

/mnt/./maillog

/mnt/./messages

/mnt/./secure

/mnt/./wtmp

/mnt/./grclean.log

/mnt/./mibmgrd.log

/mnt/./cli.log

#umountf -A

Device /dev/wd3a unmounted.

#

3.Edit the file /etc/rc.boot and see if the line mount /dev/wd3a /var/log is present; if not, add this line at the end of the file.

4.Edit the file /etc/fstab and add this line as shown in the following excerpt:

/dev/wd3a /var/log ufs rw 0 2 # PCMCIA slot A

##Each line is of the form:

##device mount_point type flags dump fsck_pass

/dev/rd0a/ ufs rw 0 0

/dev/wd3a /var/log ufs rw 0 2 # PCMCIA slot A

Installation and Configuration

77

5.Edit the file /etc/syslog.conf to specify the location where the logs will be kept. Uncomment the local log configuration lines in the “Log messages to Disk” section by removing #disk# from each line, and specify /var/log as the directory for each log. The entries should now look like the following:

*.err;*.notice;kern.debug;lpr,auth.info;mail.crit /var/log/messages cron.info /var/log/cron

local0.info /var/log/gritd.packets local1.info /var/log/gr.console local2.* /var/log/gr.boot local3.* /var/log/grinchd.log local4.* /var/log/gr.conferrs local5.* /var/log/mib2d.log

6.Check that /etc/grclean.conf and /etc/grclean.logs.conf have entries pointing to files in /var/log.

The /etc/grclean.conf file entries should look like the following:

###################################################################

#port card dump files.

###################################################################

hold=4

size=1

remove=y

local=y logfile=/var/portcards/grdump.*

###################################################################

#cleanup our own log file, if necessary.

###################################################################

DEFAULTS hold=2 local=y size=10000

logfile=/var/log/grclean.log

The /etc/grclean.logs.conf file entries should look like the following:

Installation and Configuration

78

***********************************************************************

*Log files that used to be archived by the /etc/{daily|weekly|monthly}

*scripts.

***********************************************************************

size=150000

logfile=/var/log/gr.console

size=11000

logfile=/var/log/gr.boot

7.Save all changes and reboot:

#grwrite -v

#reboot

8.After the SP Switch Router is up and running again, use csconfig -a to verify that the PCMCIA interface is available and the PCMCIA disk are up.

For a quick test, run grconslog -vf. If the setup is correct, grconslog will not complain about a missing /var/log/gr.console file, but instead will show all entries and stay up running, giving updates of new entries to the file onto the screen. To stop grconslog, use Ctrl+C.

3.4 Attaching SP Switch Router Cables

Three types of cables must be attached:

The administrative Ethernet LAN cable

The SP Switch Router Adapter card to SP Switch cable(s)

The ground strap to the SP frame

3.4.1 Ethernet Cable

Route the Ethernet twisted-pair cable between the SP Switch Router unit and the Ethernet hub, then connect the cable to the SP Switch Router control board and to the Ethernet hub. See Figure 31 on page 80 on how this might be accomplished.

Installation and Configuration

79

SP Control Workstation

Hub

SP Switch Router

Administrative Ethernet network

Control board

Figure 31. SP System Administrative Ethernet Connections

3.4.2 SP Switch Cable

The SP Switch Router Adapter card provides one full-duplex attachment and requires a specific cable with 50-pin connector ends, obtainable from IBM. The cable has a unique signal wiring map, and is not replaceable by a 50-pin HSSI cable, for example. SP Switch Router Adapter card cables are available in 10and 20-meter lengths (32 or 65 feet). Excess cable lengths should be bound in a figure-eight pattern. Do not wind excess cable into circular coils.

Each connector end has 50 fragile pins. Pins can become bent when making the connection to the media card if alignment is wrong. If an SP Switch Router Adapter card link does not work after cabling, check both ends of the cable for bent pins. When not connected, keep the plastic caps on the ends.

3.4.3 Procedure for Connecting Cards to the SP Switch

This procedure connects the SP Switch Router Adapter card(s) to the SP Switch. Before the SP Switch Router unit can begin full operation, all other router media cards must be configured with appropriate customer configuration information.

Make sure you have labeled the SP Switch cable to show which media card and SP Switch port it will be connected to. Keep in mind that for any work done on the SP Switch you should have shut down and powered off the SP System and also turned off the central power supply switch at the left front edge of the SP.

Execute the following steps to make the connections:

1.If there are any terminators on the media card or the switch assembly where you need to attach the switch cable, remove them now.

Installation and Configuration

80

2.Using appropriate frame entry and exit holes for cable management, route the SP Switch cable between the SP Switch Router unit and the SP Switch.

3.Connect the SP Switch cable to both the media card and the correct SP Switch port, as follows:

Connection to media card

The EMI shielding fitted inside the connector end can make insertion difficult, so be sure to insert the connector end as perpendicular as possible. (Pins can be damaged when the connector is inserted at too much of an angle.) Seat the connector firmly so the spring clips engage.

Connection to SP Switch port

The cable ends should click onto the connectors. Determining the correct Switch port is described in Section 3.5.1, “Determining the Switch Connection for a Dependent Node” on page 82.

4.Make sure both ends of the cable are firmly seated by pulling on them lightly.

At this point, the SP Switch Router Adapter card configuration information must be entered on the SP CWS to enable the PSSP code and SP Switch to recognize the adapter. These tasks are discussed in Section 3.5, “Configuration Required on the SP System” on page 81.

3.5 Configuration Required on the SP System

This section describes the SP Switch Router-related configuration information that should be defined by the SP administrator and then entered from the SP CWS before the SP Switch Router Adapter card is configured.

The SP Switch Router-related configuration information includes the following:

The SP Switch Router Ethernet IP address

The SP Switch Router Ethernet hostname (that is, the SP Switch Router’s administrative Ethernet hostname)

Unique node numbers for SP Switch Router Adapter cards

The SP Switch Router Adapter card configuration information enables the PSSP code and the SP Switch to recognize and communicate with this card.

Installation and Configuration

81

3.5.1 Determining the Switch Connection for a Dependent Node

The SP Switch Router Adapter connection replaces an SP node connection to the SP Switch. Each SP Switch Router Adapter media card is referred to as a dependent node, and is assigned a node number that corresponds to its specific connection on the SP Switch.

The node number is determined by the SP system administrator based on an understanding of how node numbers are assigned in the SP System and the rules for choosing a valid, unused SP Switch port.

See Figure 32 on page 83 for a brief overview of how Switch port numbers are assigned and how to look for the best suited port number for the actual scenario.

Note: Look out for the change in numbering regarding Frame n+2 and Frame n+3!

See Figure 33 on page 84 on how node numbers are assigned independent of whether the frame is fully populated.

If proper planning has been done to assign the node number, the system administrator knows which SP frame, Switch board, and node slot corresponds to a dependent node. Given this information, you can determine which jack on the Switch board should be used by consulting the Switch Cable Charts for the SP Switch in RS/6000 SP: Maintenance Information, Volume 1, Installation and Customer Engineer Operations, GC23-3903.

Do not attempt to connect an SP Switch Router Adapter to the SP Switch until proper planning has been done to assign the node number.

Once the node number is assigned, the SP system administrator can define the corresponding dependent node using SMIT as described in the "Managing Extension Nodes" section of RS/6000 SP: Administration Guide Version 2 Release 4, GC23-3897.

After defining new dependent nodes on the SP, the administrator should use the Eannotator command to annotate the SP Switch topology file. With the file annotated, even if the administrator is not sure of the frame, switch board, or node slot for the dependent node, you can determine the corresponding switch connection with the procedure described in Section 3.5.2, “Procedure to Get the Jack Number” on page 84.

Installation and Configuration

82