IBM 9077 SP User Manual
IBM 9077 SP Switch Router:
Get Connected to the SP Switch
Hajo Kitzhöfer, Steffen Eisenblätter, Uwe Untermarzoner
International Te chnic al Suppor t Organization
http://www.redbooks.ibm.com
SG24-5157-00
International Technical Support Organization
IBM 9077 SP Switch Router:
Get Connected to the SP Switch
November 1998
SG24-5157-00
Take Note!
Before using this information and the product it supports, be sure to read the general information in
Appendix D, “Special Notices” on page 305.
First Edition (November 1998)
This edition applies to PSSP Version 2, Release 4 for use with AIX 4.3.1 and Ascend Embedded/OS
Version 1.4.6.4.
Comments may be addressed to:
IBM Corporation, International Technical Support Organization
Dept. HYJ Mail Station P099
522 Sou th Road
Poughkeepsie, New York 12601-5400
When you send information to IBM, you grant IBM a non-exclusive right to use or distribute the
information in any way it believes appropriate without incurring any obligation to you.
© Copyright International Business Machines Corporat ion 1998 . All rights reserved
Note to U.S Government Users – D oc umen tation r elated to restricted r ights – Use, duplic ation or dis clos ure is
subject to restrictions set for th in GS A ADP S che dule Contr act with IBM C orp.
Contents
Figures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .ix
Tables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xi
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xiii
The Team That Wrote This Redbook. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xiii
Comments Welcome . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv
Part 1. Introd ucin g a n d I ns talling the GRF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Chapter 1. Dependent Node. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1 Dependent Node Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Limitations of the Dependent Node. . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Chapter 2. Router Node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5
2.1.1 Moti v a tion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1.2 Des ig n Objectives. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.1.3 Wh a t is a R o u ter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.1.4 Routing w ithout t h e G R F . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.1.5 Routing w ith the G R F . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.1.6 Overview of Supported Routing Protocols. . . . . . . . . . . . . . . . . . 15
2.1.7 Media Adapters At-a-Glance. . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.1.8 Benefits of the GRF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.1.9 Price Comp arison. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2 GRF Sof tware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2.1 IP Protoco l . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.2.2 Supported Routing Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.2.3 Filte r i n g . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.2.4 System Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.3 GRF Ha rdware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.3.1 GR F Block Diagra m . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.3.2 GR F F e a t u r e s. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.3.3 IP Switch and Control Board. . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2.3.4 Memory Guidelines for the IP Switch Control Board . . . . . . . . . . 35
2.3.5 Char a cterist i c s of GRF Med i a C a r d s . . . . . . . . . . . . . . . . . . . . . . 36
2.3.6 SP Switch Router Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
2.3.7 Med ia Card Perfor ma n ce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
2.3.8 Othe r Med ia C a r d s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.3.9 GR F O p e r a ting E n vironm ent . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.4 PSSP Enhancements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
© Copyright IBM Corp. 1998 iii
2.4.1 SDR Enhancements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.4.2 New Co mmand s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
2.4.3 Enhanced Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
2.4.4 Har dware Perspe c ti v e s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
2.4.5 SP Extension Node SNMP M anager. . . . . . . . . . . . . . . . . . . . . . 58
2.4.6 Dependent Node MIB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
2.4.7 Coexisten c e . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
2.4.8 Par titionin g . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
2.5 Planning for the GRF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
2.6 Planning for the Dependent Node. . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
2.7 C o n c l usi o n . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
Part 2. Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
Chapter 3. Install ation and Config urat ion. . . . . . . . . . . . . . . . . . . . . . . 69
3.1 Ini t i a l Confi g u ration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
3.2 P re - In s ta llation Assum p ti o n s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
3.2.1 Orde r o f Informatio n . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
3.3 Installing an SP Switch Router Adapter Card . . . . . . . . . . . . . . . . . . . 75
3.3.1 Inst a l la ti on Overv i e w . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
3.3.2 Installing the PCMCIA Spinning Disk . . . . . . . . . . . . . . . . . . . . .76
3.4 A tta ching SP Sw i tc h R o u te r Cabl e s . . . . . . . . . . . . . . . . . . . . . . . . . . 79
3.4.1 Ethe r n e t Cable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .79
3.4.2 SP Switc h Cable. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
3.4.3 Procedure for Connecting Cards to the SP Switch . . . . . . . . . . . 80
3.5 Configuration Required on the SP System . . . . . . . . . . . . . . . . . . . . . 81
3.5.1 Determining the Switch Connection for a Dependent Node. . . . . 82
3.5.2 Procedure to Get the Jack Number. . . . . . . . . . . . . . . . . . . . . . . 84
3.6 Multiple Frames for Multiple System Connections . . . . . . . . . . . . . . . 85
3.7 S te p - b y- Step M e d ia C a r d Confi g u ration . . . . . . . . . . . . . . . . . . . . . . . 86
3.7.1 Configuration Files and Their Uses. . . . . . . . . . . . . . . . . . . . . . . 86
3.8 Step 1. Check SNMP in the SP Switch Router System. . . . . . . . . . . . 88
3.9 Put SNMP Changes into Effect. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
3.10 Step 2. Assig n IP Ad d resses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
3.10.1 Method 1: Use SP SNMP M anager - Recommended . . . . . . . . 89
3.10.2 Method 2: Edit /etc/grifconfig.conf - Optional . . . . . . . . . . . . . . 93
3.10.3 Putt i n g g r ifconfi g .c o n f Additi o n s into E ff e ct. . . . . . . . . . . . . . . .95
3.11 Step 3. Change Profile Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
3.12 Step 4. Run d e v 1 config . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
3.13 Step 5. Rese t SP Switch R o u ter System to In stall F il e s . . . . . . . . . . 96
3.13.1 Sav i n g Configu r a tion Fil e s . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
3.14 Verify an SP Switch Router Adapter Card on the Router . . . . . . . . . 97
3.14.1 Verify Media Card Operation Using ping. . . . . . . . . . . . . . . . . . 97
iv IBM 9077 SP Switch Router: Get Co nnec ted to the SP S witch
3.14.2 Check Media Card Status Using grcard . . . . . . . . . . . . . . . . . . 98
3.14.3 Res e t M e d ia Card U si ng grres e t. . . . . . . . . . . . . . . . . . . . . . . . 99
3.14.4 Usin g g r stat to Displa y GR F Statist i c s . . . . . . . . . . . . . . . . . . . 99
3.15 Bringing the SP Switch Router Adapter Card Online with the SP . . 100
3.15.1 Checking Connectivity to the SP System . . . . . . . . . . . . . . . . 101
Chapter 4. Conf igu ration of IP-Forw ardin g M e di a C ards . . . . . . . . . . 105
4.1 Ethernet 10/100Base-T Configuration. . . . . . . . . . . . . . . . . . . . . . . . 105
4.1.1 Physical and Logical Interfaces . . . . . . . . . . . . . . . . . . . . . . . . 105
4.1.2 Configuration File and Profile Overview . . . . . . . . . . . . . . . . . . 106
4.1.3 Installing Configurations or Changes . . . . . . . . . . . . . . . . . . . . 107
4.1.4 Assi g n IP Add resses - g r ifconfig.conf . . . . . . . . . . . . . . . . . . . . 107
4.1.5 Specify Ethernet Card Parameters . . . . . . . . . . . . . . . . . . . . . . 108
4.1.6 Some maint Commands f or the Ethernet M edia Cards . . . . . . . 109
4.2 A T M OC-3 c Config u r a ti o n . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
4.2.1 Physical and Logical ATM Interfaces . . . . . . . . . . . . . . . . . . . . 110
4.2.2 Installing Configurations or Changes . . . . . . . . . . . . . . . . . . . . 113
4.2.3 Configuration Files and Profiles . . . . . . . . . . . . . . . . . . . . . . . . 113
4.2.4 Assi g n IP Add resses - g r ifconfig.conf . . . . . . . . . . . . . . . . . . . . 114
4.2.5 Spe c i fy ATM C a r d P a ra meters . . . . . . . . . . . . . . . . . . . . . . . . . 115
4.2.6 Configuri ng PVCs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
4.2.7 Some maint Commands for the ATM O C-3c Media Card . . . . . 116
4.2.8 Usin g g r rt to D is p l a y the Ro u te Table . . . . . . . . . . . . . . . . . . . . 118
4.2.9 Usin g g r stat to Displa y GR F Statis ti c s . . . . . . . . . . . . . . . . . . . 119
4.3 A T M OC-1 2 c Configu r a ti o n . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
4.3.1 Physical and Logical ATM Interfaces . . . . . . . . . . . . . . . . . . . . 119
4.3.2 Installing Configurations or Changes . . . . . . . . . . . . . . . . . . . . 120
4.3.3 Configuration Files and Profiles . . . . . . . . . . . . . . . . . . . . . . . . 120
4.4 FD DI Con fi g u r a tio n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
4.4.1 Separate Networks versus Bridging . . . . . . . . . . . . . . . . . . . . . 126
4.4.2 Nam i n g t h e F D D I Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
4.4.3 Phy sical In te r fa ce Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
4.4.4 GR F Interfa ce Name. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
4.4.5 Configuration Files and Profiles . . . . . . . . . . . . . . . . . . . . . . . . 128
4.4.6 Assi g n IP Add resses - g r ifconfig.conf . . . . . . . . . . . . . . . . . . . . 129
4.4.7 Spe c i fy F D D I Card Parame te rs. . . . . . . . . . . . . . . . . . . . . . . . . 130
4.4.8 Installing Configurations or Changes . . . . . . . . . . . . . . . . . . . . 130
4.4.9 Some maint Commands for the FDDI Media Card . . . . . . . . . . 131
4.4.10 Usin g g r r t to Dis p l a y the Ro u te T a b le . . . . . . . . . . . . . . . . . . . 132
4.4.11 Usin g g r stat to Displa y GR F Statist i c s . . . . . . . . . . . . . . . . . . 133
4.5 H IP PI Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
4.5.1 Introduction to HIPPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
4.5.2 HIPPI Co n fig u r a ti on Option s. . . . . . . . . . . . . . . . . . . . . . . . . . . 138
v
4.5.3 Physical and Logical Interfaces . . . . . . . . . . . . . . . . . . . . . . . . 139
4.5.4 Configuration Files and Profiles . . . . . . . . . . . . . . . . . . . . . . . . 140
4.5.5 Installing Configurations or Changes . . . . . . . . . . . . . . . . . . . . 141
4.5.6 Some maint Commands f or the HIPPI Medi a Card . . . . . . . . . . 141
4.6 C o n fig u r i n g Bridg in g. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
4.6.1 GR F Bridg in g Implementatio n. . . . . . . . . . . . . . . . . . . . . . . . . . 142
4.6.2 Simultaneous Routing and Bridging . . . . . . . . . . . . . . . . . . . . . 143
4.6.3 Configurat i o n Option s. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
4.6.4 Interoperability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
4.6.5 Spanning Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
4.6.6 Bridge Filtering Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
4.6.7 Frag mentat i o n . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
4.6.8 Spa mming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
4.6.9 Bridging Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
4.6.10 Management Tools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
4.6.11 Configuration File and Profile Overview . . . . . . . . . . . . . . . . . 148
4.6.12 Brid g i n g A T M . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
4.6.13 Brid g i n g FD DI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
4.6.14 Brid g i n g E th e r n e t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
Chapter 5. Single RS/6000 SP and Single SP Switch Router. . . . . . . 157
5.1 Single SP Partition and Single SP Switch Router Adapter Card . . . . 157
5.1.1 SP Switch - Ethernet Connection . . . . . . . . . . . . . . . . . . . . . . . 157
5.1.2 SP Switch - FDDI Connection. . . . . . . . . . . . . . . . . . . . . . . . . . 162
5.1.3 SP Switch - ATM Connection . . . . . . . . . . . . . . . . . . . . . . . . . . 167
5.1.4 SP Switch - FDDI Connection (Distinct FDDI Networks) . . . . . . 174
5.1.5 SP Switch - FDDI Connection in an ADSM Environment. . . . . . 185
5.2 Single SP Partition and Multiple SP Switch Router Adapter Cards . . 187
5.2.1 Configuration of a Dual SP Switch Router Connection . . . . . . . 187
5.2.2 Com p l e x C o n fi g u r a tio n . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
5.2.3 Recovery Procedure for an SP Switch Adapter Card Failure. . . 196
5.3 Multiple SP Partition and Multiple SP Switch Router Adapter Cards. 197
Chapter 6. Multiple RS/6000 SPs and One SP Switch Router . . . . . . 203
6.1 RS/6000 SP Switch - RS/6000 SP Switch Connection . . . . . . . . . . . 203
6.2 S h a rin g Networ k Resou r ces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
Chapter 7. Multiple RS/6000 SPs and Multiple GRFs . . . . . . . . . . . . . 209
7.1 ATM OC-3c Backbone Connection. . . . . . . . . . . . . . . . . . . . . . . . . . 209
7.1.1 ATM OC-3c Backbone - Using One P ort. . . . . . . . . . . . . . . . . . 210
7.1.2 ATM OC-3c Backbone - Using Two P orts . . . . . . . . . . . . . . . . . 215
7.2 ATM OC-12c Backbone - One Port. . . . . . . . . . . . . . . . . . . . . . . . . . 222
7.3 HIPPI Backbone Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
vi IBM 9077 SP Switch Router: Get Connected to the SP Switch
Appendix A. Laboratory Hardware and Software Conf igura tion . . . . 233
A.1 Node and Control Workstation Configuration . . . . . . . . . . . . . . . . . . . . . 233
A.1.1 Hard Disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
A.1.2 Software Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
A.1.3 Network Options and Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
A.2 SP Switch Pool Size Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
A.3 7025-F50 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
A.4 SP IP Switch Router Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
Appendix B. GRF Configuration File s . . . . . . . . . . . . . . . . . . . . . . . . . . 261
B.1 /root/.profile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
B.2 /etc/Release . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
B.3 /etc/bridged.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
B.4 /etc/fstab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
B.5 /etc/grarp.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
B.6 /etc/gratm.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268
B.7 /etc/grclean.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
B.8 /etc/grclean.logs.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
B.9 /etc/grdev1.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
B.10 /etc/grifconfig.conf. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
B.11 /etc/grlamap.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
B.12 /etc/grroute.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
B.13 /etc/hosts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286
B.14 /etc/inetd.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286
B.15 /etc/motd. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
B.16 /etc/rc.local . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
B.17 /etc/snmpd.conf. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288
B.18 /etc/syslog.conf. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
B.19 /etc/ttys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292
Appendix C. Hardware and Software Information . . . . . . . . . . . . . . . . 295
C.1 The Front Panel of the SP Switch Router Adapter Card - Operational. . 295
C.2 SP Switch Router Adapter Media Card LEDs. . . . . . . . . . . . . . . . . . . . . 296
C.3 SP Switch Router Adapter Media Card - Bootup . . . . . . . . . . . . . . . . . . 297
C.4 Connectors and Receptacles for Different Media . . . . . . . . . . . . . . . . . . 298
C.5 Chip Interconnection on the TBS Board. . . . . . . . . . . . . . . . . . . . . . . . . 298
C.6 Updating Router Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
C.6.1 The SP Switch Router as an IBM Product . . . . . . . . . . . . . . . . . . . 299
C.6.2 Obtaining New Machine Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
C.6.3 Support for Code Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
C.6.4 Sample Steps to Upgrade the System Software . . . . . . . . . . . . . . 300
C.6.5 Sample Execution of grf_update Script . . . . . . . . . . . . . . . . . . . . . 301
vii
Appendix D. Special Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
Appendix E. R e l ated P ubl ica tion s. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
E.1 International Technical Support Organization Publi catio ns . . . . . . . . . . 309
E.2 Redbooks on CD-ROMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
E.3 Other Publications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
How to Get ITS O R edbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311
How IBM Employees Can Get ITSO Redbooks. . . . . . . . . . . . . . . . . . . . . . . 311
How Customers Can Get ITSO Redbooks. . . . . . . . . . . . . . . . . . . . . . . . . . . 312
IBM Redbook Order Form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
List of Abbrevi a tions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
ITSO Redbook Evaluat ion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
viii IBM 9077 SP Switch Router: Get Connec ted to the SP S witch
Figures
1. SP Switch Router. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2. Functional Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3. Typical Router Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4. Table-Based Routing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
5. Routing without GRF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
6. Routing with GRF. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
7. GRF 400 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
8. Conventional Routers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
9. Switched Routers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
10. Price Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
11. GRF Model s. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
12. GRF Architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
13. Data Packet Transfer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
14. Routing Packet Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
15. Side View of GRF 400 Chassis with Slots Numbered . . . . . . . . . . . . . . . . 32
16. Top View of the GRF 1600 Chassis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
17. IP Switch Control Board. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
18. System RAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
19. SP Switch Router Adapter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
20. Hardware Perspectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
21. Action Menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
22. Hardware Notebook. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
23. System Partition Aid Perspectives. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
24. System Partition Aid Notebook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
25. Coexistence. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
26. Partitioning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
27. The Labo r a to ry Ha r d w a r e Insta llati o n . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
28. Connecting the GR F to the SP Switch and the CWS . . . . . . . . . . . . . . . . 69
29. Connecting the GR F to the Frame. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
30. Connecting the GR F Consol e . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
31. SP System Admini strative Ethernet Connections . . . . . . . . . . . . . . . . . . . 80
32. Switch Port Assignm ent s in Supported Fram e Configurations . . . . . . . . . 83
33. Node Numbering for an SP System. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
34. How Frames Enab le Connections to Multiple SP Switches. . . . . . . . . . . . 86
35. Component s in the SP Switch Router Adapter Card’s Interface Name. . . 93
36. Component s of the Ethernet Interface Name . . . . . . . . . . . . . . . . . . . . . 106
37. ATM OC-3c Physical and Logi cal Interfaces . . . . . . . . . . . . . . . . . . . . . . 110
38. Component s in the ATM OC-3c Interface Name . . . . . . . . . . . . . . . . . . . 111
39. Component s Form ing a Virtual Path . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
40. ATM OC-12c Physical and Logical Interfaces . . . . . . . . . . . . . . . . . . . . . 120
© Copyright IBM Corp. 1998 ix
41. Master/Slave Connec tors for SAS Interfaces . . . . . . . . . . . . . . . . . . . . . 122
42. A/B Connect ors for DAS Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
43. Allowed SAS and DAS Configurations. . . . . . . . . . . . . . . . . . . . . . . . . . . 123
44. Optical Bypass Switch Attachments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
45. Dual Homing Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
46. Assigning Numbers to FDDI Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . 127
47. Physical Interface Numbering on the FDDI Media Card . . . . . . . . . . . . . 128
48. GRF Interface Name for FDDI Interfaces . . . . . . . . . . . . . . . . . . . . . . . . 128
49. HIPPI I-Field Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
50. Component s in the HIPPI Interface Name. . . . . . . . . . . . . . . . . . . . . . . . 139
51. Interface Name for FDDI, Ethernet and ATM OC-3c Interfaces . . . . . . . 150
52. One Card - One SP Partition Sample Configuration . . . . . . . . . . . . . . . . 157
53. SP Switch - Ethernet Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
54. SP Switch - FDDI Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
55. SP Switch - ATM Connection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
56. SP Switch - FDDI Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
57. SP Switch - FDDI Connection (Bridging) . . . . . . . . . . . . . . . . . . . . . . . . . 180
58. SP Switch Router in an ADSM Environment . . . . . . . . . . . . . . . . . . . . . . 185
59. Connecting On e SP Switch with Two SP Switch Router Adapter Cards. 187
60. Configuration with Dual SP Switch Router - SP Switch Connection . . . . 190
61. IP Traffic Flow When Issuing ping 192.168.13.1 on Node 6 . . . . . . . . . . 195
62. IP Traffic Flow When Issuing ping 192.168.13.1 on Node 10 . . . . . . . . . 195
63. IP Traffic Flow When Issuing ping 192.168.13.1 on Node 8 . . . . . . . . . . 196
64. Partition-to-Partition Connection with an SP Switch Router . . . . . . . . . . 198
65. Two RS/6000 SPs Connec ted to GRF 1600 . . . . . . . . . . . . . . . . . . . . . . 203
66. Sharing Network Resou rces between Two SP s . . . . . . . . . . . . . . . . . . . 207
67. Connection of Two SPs with Two SP Switch Routers . . . . . . . . . . . . . . . 209
68. SP Switch - ATM - SP Switch Connection. . . . . . . . . . . . . . . . . . . . . . . . 211
69. SP Switch - ATM Bridged - SP Switch Connection . . . . . . . . . . . . . . . . . 215
70. SP Switch - ATM OC-12c - SP Switch Connection . . . . . . . . . . . . . . . . . 223
71. SP Switch - HIPPI - SP Switch Connection. . . . . . . . . . . . . . . . . . . . . . . 228
72. Front Panel of the SP Switch Router Adapter Card with LEDs . . . . . . . . 295
73. The SP Switch Board. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
x IBM 9077 SP Switch Router: Get Connected to the S P Swi tch
Tables
1. Memory Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2. DependentNode Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3. DependentAdapter Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
4. Additional SDR Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
5. New Commands (root Executable) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
6. New Commands (User Executable). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
7. endefnode Command Opt ion s. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
8. enrmnode Command Opt ions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
9. endefadapter Comma nd Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
10. enadmin Com m and Opt ions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
11. splstnode Com man d Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
12. splstadapt er Command Opt ion s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
13. Enhanced Com ma nds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
14. Configuration of SP Switch - Ethernet Connection . . . . . . . . . . . . . . . . . 159
15. Configuration of an SP Switch - FDDI Connection . . . . . . . . . . . . . . . . . 163
16. Configuration of SP Switch - ATM Connection . . . . . . . . . . . . . . . . . . . . 168
17. Configuration of SP Switch - FDDI Connection . . . . . . . . . . . . . . . . . . . . 175
18. Configuration of SP Switch - FDDI Connection (Bridging). . . . . . . . . . . . 181
19. Configuration of a Dual SP Switch Router Connection . . . . . . . . . . . . . . 187
20. Configuration of a Dual SP Switch Router - SP Switch Connection . . . . 191
21. Configuration of a Partition - Partition Connection . . . . . . . . . . . . . . . . . 199
22. Configuration of SP Switch - SP Switch Connection . . . . . . . . . . . . . . . . 204
23. Configuration of SP Switch - ATM - SP Switch . . . . . . . . . . . . . . . . . . . . 212
24. Configuration of SP Switch - ATM Bridged - SP Switch . . . . . . . . . . . . . 216
25. Configuration of SP Switch - ATM OC-12c - SP Switch . . . . . . . . . . . . . 224
26. Configuration of SP Switch - HIPPI - SP Switch . . . . . . . . . . . . . . . . . . . 228
27. Configuration of SP 21. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
28. Configuration of SP 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
29. Hard Disk Equipment of SP 21 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
30. Hard Disk Equipment of SP 2 Part 1 of 2. . . . . . . . . . . . . . . . . . . . . . . . . 237
31. Hard Disk Equipment of SP 2 Part 2 of 2. . . . . . . . . . . . . . . . . . . . . . . . . 238
32. Software Levels on CWS and All Nodes Part 1 of 14 . . . . . . . . . . . . . . . 239
33. Software Levels on CWS and All Nodes Part 2 of 14 . . . . . . . . . . . . . . . 240
34. Software Levels on CWS and All Nodes Part 3 of 14 . . . . . . . . . . . . . . . 241
35. Software Levels on CWS and All Nodes Part 4 of 14 . . . . . . . . . . . . . . . 242
36. Software Levels on CWS and All Nodes Part 5 of 14 . . . . . . . . . . . . . . . 243
37. Software Levels on CWS and All Nodes Part 6 of 14 . . . . . . . . . . . . . . . 244
38. Software Levels on CWS and All Nodes Part 7 of 14 . . . . . . . . . . . . . . . 245
39. Software Levels on CWS and All Nodes Part 8 of 14 . . . . . . . . . . . . . . . 246
40. Software Levels on CWS and All Nodes Part 9 of 14 . . . . . . . . . . . . . . . 247
© Copyright IBM Corp. 1998 xi
41. Software Levels on CWS and All Nodes Part 10 of14. . . . . . . . . . . . . . . 248
42. Software Levels on CWS and All Nodes Part 11 of 14 . . . . . . . . . . . . . . 249
43. Software Levels on CWS and All Nodes Part 12 of 14 . . . . . . . . . . . . . . 250
44. Software Levels on CWS and All Nodes Part 13 of 14 . . . . . . . . . . . . . . 251
45. Software Levels on CWS and All Nodes Part 14 of 14 . . . . . . . . . . . . . . 252
46. Network Options of CWS and All Nodes Part 1 of 3 . . . . . . . . . . . . . . . . 253
47. Network Options of CWS and All Nodes Part 2 of 3 . . . . . . . . . . . . . . . . 254
48. Network Options of CWS and All Nodes Part 3 of 3 . . . . . . . . . . . . . . . . 255
49. Network Op tions of 7025-F50 Part 1 of 3 . . . . . . . . . . . . . . . . . . . . . . . . 256
50. Network Op tions of 7025-F50 Part 2 of 3 . . . . . . . . . . . . . . . . . . . . . . . . 257
51. Network Op tions of 7025-F50 Part 3 of 3 . . . . . . . . . . . . . . . . . . . . . . . . 258
52. SP Switch Router Adapter Media Card LEDs . . . . . . . . . . . . . . . . . . . . . 296
53. SP Switch Rou te r Adapter Me d ia Card L EDs - RX/TX . . . . . . . . . . . . . . 29 6
54. SP Switch Router Adapter Media Card LEDs During Bootup . . . . . . . . . 297
55. Media Card Cables and Connectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
xii IBM 9077 SP Switch Router: Get Connected to the S P S witch
Preface
The GRF is a high-performance switched IP Router which provides
high-speed data communication links between IBM RS/6000 SP and external
networks or hosts. It acts as a special-purpose SP node that routes IP traffic
between SP nodes on the SP Switch and the out side world. Connected
directly to the SP Switch, the router offers significantly improved SP I/O
performance. When packaged with an IBM SP system, the GRF router is
referred to as the SP Switch Router.
This redbook helps you i nstal l , tailor and configure the SP Swi tch Router, I BM
machine type 9077. The SP Switch Router is also known as the "Gigarouter"
or High Performance Gateway Node (HPGN).
The first part of the book gives an overview of the GRF architecture and how
the router was integrated into the SP. It emphasizes the advantages of
choosing a dedicated router node in some configurations, as opposed to
using standard nodes for the routing task. This part also describes some
routing fundamentals, particularly focusing on concepts like IP- and
switch-routing.
The second part presents sample configurations that were carefully chosen to
match frequently occurring customer situations. The basic configurations
shown are building blocks for more complex networking topologies that
include the SP Switch Router and may inspire more sophisticated
configurations. All configurations described were tested and provide some
comparable performance figures.
This publication is intended to give IBM customers, system engineers, and
marketing personnel a br oad understanding of this new arch itec ture and what
it is used for.
The Team That Wrote This Redbook
This redbook was produced by a team of specialists from around the world
working at the International Technical Support Organization, Poughkeepsie
Center.
Dr Hajo Kitzhöfer is an Advisory International Technical Support
Organization (ITSO) Specialist for RS/6000 SP at the Poughkeepsie Center.
He holds a Ph.D. degree in electrical engineering from the Ruhr-University of
Bochum (RUB). Before joining ITSO, he worked as an SP Specialist at the
RS/6000 and AIX Competence Center, IBM Germany. He has worked at IBM
© Copyright IBM Corp. 1998 xiii
for eight years. His areas of expertise include RS/6000 SP, SMP, and
Benchmarks. He now specializes in SP System Management, SP
Performance Tuning and SP hardware.
Dr Steffen Eisenblätter is an AIX Software Specialist in the RS/6000 SP
Software Support Center, Germany. He holds a Ph.D. degree in physics from
the University of Leipzig. He joined IBM in 1997 and has focused on RS/6000
SP products and TCP/IP.
Uwe Untermarzoner is an RS/6000 SP Technical Support Specialist with
IBM Germany. He joined IBM 1989. He has ten y ears of experience in AIX
and five years of experience with the SP, mostly in the commercial
environment. He joined IBM at 1989. His areas of expertise include AIX,
RS/6000 SP, SMP, PSSP, Networking, Performance Tuning and Systems
Management.
Thanks to the following people for their invaluable contributions to this
project:
Ronald Linton
IBM PPS Lab Poughkeepsie
Gene Novitsky
Ascend Communications, Inc.
Frank May
IBM Worldwide RS/6000 SP Product Marketing
Wes Kinard
IBM RS/6000 Networking Technologies
Marcelo R. Barrios
International Technical Support Organization, Poughkeepsie Center
xiv IBM 9077 SP Switch Router: Get Con nected to the SP Switc h
Comments Welcom e
Your comments are important to us!
We want our redbooks to be as helpful as possible. Please send us your
comments about this or other redbooks in one of the following ways:
• Fax the evaluation form found in “ITSO Redbook Evaluation” on page 323
to the fax number shown on the form.
• Use the electronic evaluation form found on the Redbooks Web sites:
For Internet users
http://www.redbook s.ibm.com
For IBM Intranet usershttp://w3.itso.ibm .com
• Send us a note at the following address:
redbook@us.ibm. com
xv
xvi IBM 9077 SP Switch Router: Get Con nected to the SP Switc h
Part 1. Introducing and I nstalling the GRF
© Copyright IBM Corp. 1998 1
2 IBM 9077 SP Switch Router: Get Connected to the S P Swi tch
Chapter 1. Dependent Node
This chapter provides an overview of a
start by defining the dependent node and the rationale behind its design.
1.1 Dependent Node Architecture
The Dependent Node Architecture refers to a processor or node, possi bl y not
provided by IBM, for use with the RS/6000 SP.
Since a dependent node may not be a regular RS/6000 SP node, not all the
functions of a node can be performed on it, which is why it is called
"dependent". For example, it does not allow all the functions of the fault
service (Worm) daemon, as other RS/6000 SP nodes with access to the SP
Switch do.
The objective of this architecture i s to allow the other processors or hardware
to easily work together with the RS/6000 SP, extending the scope and
capabilities of the system.
The dependent node connects to the RS/6000 SP Switch (but not to the
earlier High Performance Switch, HiPS).
The SP Switch Router Adapter is the first product to exploit the Dependent
Node Architecture.
dependent node
in RS/6000 SP. We
1.2 Limitations of the De pendent N ode
The following are limitations associated with use of the dependent node:
• To use the dependent node in an RS/6000 SP requires the SP Extension
Node SNMP Manager to be installed in the Control Workstation. The SP
Extension Node SNMP Manager requires UDP port 162 in the Control
Workstation. Other SNMP managers, such as Netview, also require this
port. To allow the two SNMP managers to coexist, the SP Extension Node
SNMP Manager must use an alternative UDP port.Dependent nodes are
not allowed in Node Groups.
• Only the 8-port and 16-port SP Switch are supported. The 8-port and
16-port High Performance Switch (the old SP Switch) are not supported.
• The
© Copyright IBM Corp. 1998 3
spmon command on the RS/6000 SP is not enhanced to support
dependent nodes. Dependent nodes can only be viewed with the
perspectives command.
• The fault service daemon runs on all switch nodes in the RS/6000 SP, but
not on the dependent node. Therefore, the dependent node does not have
the full functionality of a normal RS/6000 SP Switch node.
• The dependent node requires the SP Switch’s primary node to compute its
switch routes. Therefore, the primary node must have at least PSSP 2.3
installed, otherwise the dependent node cannot work with the RS/6000
SP.
• In the RS/6000 SP, SP Switch nodes occasionally send service packets
from one node to the next to keep track of status and links. Sometimes
these packets are sent indirectly through another switch node. As the
dependent node is not a standard RS/6000 SP Switch node, it cannot be
used to forward service packets to other nodes.
4 IBM 9077 SP Switch Router: Get Connected to the S P Swi tch
Chapter 2. Router N ode
The first dependent node is actually a new SP Switch Router Adapter in a
router. This chapter offers more details about the implementation.
Section 2.1, “Overview” on page 5 gives you an overview of SP Switch
Router. This is probably the best to get an impression what the GRF is good
for. Also a functional- and a price-comparison between using an RS/6000 SP
node and the SP Switch Router is included.
More details about the underlaying Software and Hardware can be found in
Section 2.2, “GRF Softw are” on page 18 and Section 2. 3, “GRF Hardware” on
page 24.
Section 2.4, “PSSP Enhancements” on page 40 describes the enhancements
in the PSSP Software for the support of the dependent node.
Some planning considerations which should be considered can found in
Section 2.5, “Planning f or the GRF” on page 63 and Section 2. 6, “Planning for
the Dependent Node” on page 65.
2.1 Overview
The purpose of the SP Switch Router Adapter is to allow the GRF ("goes
really fast"), manufactured by Ascend, to forward SP Switch IP traff i c to other
networks. The GRF was known as the High Performance Gateway Node
(HPGN) during the development of the adapter. IBM remarkets models of the
GRF that connect to the SP Switch as the SP Switch Router model 04S
(9077-04S) and model 16S (9077-16S). These models are not available
directly from Ascend.
Note: In the remainder of this book, we refer to the SP Switch Router as the
GRF.
The distinguishing feature of the GRF, when c ompared with other routers, is
that it has an SP Switch Router Adapter and therefore can connect directly to
the SP Switch (see Figure 1 on page 6).
© Copyright IBM Corp. 1998 5
IBM 9570 Disk
Array
Subsystem
SP Switch
HIPPI
Adapter
Adapter
HIPPI
Adapter
ATM
Processor
Nodes
SP
Switch
SP Switch
Adapter
SP Switch
Adapter
OC-12c
ATM
OC-3c
8-port
E-net 10/100
4-port
FDDI
ATM
Switch
SP Switch
SP System
Router
Figure 1. SP Switch Router
The RS/6000 SP software treats this adapter as an extension node. It is a
node because it takes up one port in the SP S witch and is assigned a node
number. It is described as an extension because it is not a standard RS/6000
SP node, but an adapter card that extends the scope of the RS/6000 SP.
Although the term
extension node
represents the node appearance of the
adapter, i t does not def i ne t he connec tion. An
for that purpose. Each extension node has an extension node adapter to
represent its connection to the SP Switch.
2.1.1 Motivation
A thin node, which has a single microchannel, is unable to deliver more than
about 30 MB/s to or from the SP Switch. Using a wide node, this number
increases to 65 MB/s but is still unable to provide full bandwidth to even one
HIPPI interface. It is also unable to feed 4 FDDI or 4 Ethernet 100BaseTx
cards at full bandwidth.
A 135 MHz wide node’s CPU becomes saturated at about 5000
packets/second. A 10 Mb/s Ethernet uses a maximum of 1500 byt es for a
6 IBM 9077 SP Switch Router: Get Connected to the S P Swi tch
extension node adapter
is used
packet size. This would only enable a wide node to handle approximately 7.5
MB/s of IP traffic.
Since Ascend’s business depends on keeping pace with networking
technology, they already support the major interfaces today. The 9077 will be
able to take advantage of any new interfaces that are developed in the future
as well, with no further development time or money expended.
With some interfaces requiring up to 5 slots, even a wide node can run out of
available slots. This forces additional nodes to be added even if there are no
performance limitations in the current configuration.
Since there are no hot plug capabilities with an SP node, any failure means
downtime on all interfaces configured in that node, and at times a lengthy
maintenance procedure. Redundancy is not built into the SP node’s
architecture.
These fact s a re illu s trated i n Figu re 2:
SP Node 9077
Bus
Route Table
Scalability
Throughput
Support
Shared
1 MCA per thin node
2 MCA per wide node
Centralized
Cache,Software Based
Cache hits <50% typical
Single port per card
Single CPU
Limited by shared bus
5000 pps
30 MB/s per thin node
65 MB/s per wide node
No support for:
HSSI
ATM OC12
Sonet
Multiple SP Switch Adapters
Non-blocking Crosspoint Switch
250 ns path setup
Independent lookups per card
Hardware based, <2.5 µs
150,000 route capacity per card
High per card port density
Per card
Processors
Route Tables
Lookup engines
Each card has dedicated bandwidth
Up to 130,000 pps
100 MB/s per card slot, full duplex
Support for:
multiple SP Switch interfaces
High-speed networks such as
HIPPI
Protocols
Figure 2. Functional Comparison
Router Node 7
2.1.2 Design Objectives
Because the dependent node is part of the RS/6000 SP, it had to be
packaged and assigned some roles consistent with other RS/6000 SP nodes.
Changes were made to the RS/6000 SP to incorporate management
requirements for the dependent node.
Ease of design and implementation were important objectives in the design.
These were accomplished by limiting the amount of switchcontrol prot ocol for
the dependent node.
SDR
New
dependent nodes. This was done to minimize the scope of the changes and
the exposure to side effects that dependent nodes may cause if they were
represented as standard nodes in the SDR.
(System Data Repository) classes were created to manage
2.1.3 What is a Router
One of the basic functions of the
between different networks. This is due its routing algorithm and its flexibility
to use almost any physical network below. A system that connects different
physical or logical net works and di rects traffic is termed a
older term
Again, IP routing is the passing of an IP packet from one device to another by
sending it on a physical or logical interface. routers interconnect networks so
that IP traffic can be routed between the systems in the networks, as shown
in Figure 3 on page 9.
IP gateway
Internet Protocol
is also used.
(IP) is its ability to connect
router
, although the
8 IBM 9077 SP Switch Router: Get Connected to the S P Swi tch
Network 1
Network 3
Router
Network 2
Figure 3. Typical Router Configuration
Network n
Routers help to reduce the amount of processing required on local systems,
since they perform the computation of routes to remote systems. For
example, a system can communicate with a remote system by passing the
message (or packets) to the router. The router works out how to get to the
remote system and forwards the message appropriately.
Storing routes on the system takes up memory. But because a system does
not have to store routes t o systems not i n i t s own s ubnet, the rout e table uses
less storage space and thereby frees up memory for other work.
The use of routing reduces network traffic, because routers encourage
subnetting, which creates a smaller network of systems. By having smaller
networks, network traffic congestion is reduced and overall network
performance and traffic control are improved.
A network’s routing configuration does not always require a routing protocol.
In situations where the routing information does not change, for example,
when there is only one possible route, the system administrator usually builds
the routing table manually. Some networks have no access to any other
TCP/IP networks, and therefore do not require routing tables at all. The three
most common routing configurations are:
Router Node 9
• Minimal routing
A network completely isolated from all other TCP/IP networks requires
only minimal routing. A minimal routing table is usually built by
ifconfig
when the network interfaces ar e conf igured. If your network does not have
direct access to other TCP/IP networks, and if you are not using
subnetting, this may be the only routing table you require.
• Static routing
A network with a limited number of gateways to other TCP/IP networks
can be configured with static routing. When a network has only one
gateway, a static route is the best choice. A static routing table is
constructed manually by the system administrator using the
route
command. See Figure 4. Static routing tables do not adjust to network
changes, so they work best where routes do not change.
Source Host Destination Host
Application Application
Transport
Gateway
Transport
Destination
192.168.1.0
192.168.12.0
default 192.168.12.1
Network Access Network Access Network Access
Figure 4. Table-Based Routing
192.168.12.3
192.168.12.2
192.168.12.2 192.168.12.3 192.168.1.2
192.168.12.0 192.168.1.0
Destination DestinationGateway Gateway Gateway
192.168.1.0
192.168.12.0
default 192.168.12.1
192.168.1.5
192.168.12.3
192.168.1.5
192.168.1.0
default
192.168.1.2
192.168.1.5
• Dynam ic routing
A network with more than one possible route to the same destination
should use dynamic routing. A dynamic routing table is built from the
information exchanged by the routing protocols. The protocols are
designed to distribute information that dynamically adjusts routes to reflect
changing network conditions. Routing protocols handle complex routing
10 IBM 9077 SP Switch Router: Get Connec ted to the SP S witch
situations more quickly and accurately than a system administrator can
do. Routing protocols are designed not only to switch to a backup route
when the primary route becomes inoperable; they are also designed to
decide which is the "best" route to a destination. On any network where
there are multiple paths to the same destination, a dynamic routing
protocol should be used.
2.1.4 Routing without the GRF
Before the GRF was available, there were only two ways to get IP traffic from
remote systems to reach the RS/6000 SP nodes:
1. By putting an addit ional IP adapter into every RS/6000 SP node.
2. By designating one or two nodes to act as a router (as shown in Figure 5).
Node
Router
Internet/Intranet
Node
SP Switch
Node
. . .
ATM
FDDI
Node
Ethernet
Figure 5. Routing without GRF
The first option was usually not chosen because it was too costly for the
following reasons:
• For systems with a large number of nodes having multiple IP adapters for
each RS/6000 SP node can be expensive.
• The number of I/O slots in the RS/6000 SP node is limited. In addition,
these slots are required to perform other tasks for the system, such as
connecting to disk or tape. Using these I/O slots to connect IP adapters
restricts the functions of the RS/6000 SP node.
Router Node 11
The second case has proven to be very expensive as well. The RS/6000 SP
node was not designed for routing. It is not a cost- effective way to route tr af fic
for the following reasons:
• It takes many CPU cycles to process routing. The CPU is not a dedicated
router and is very inefficient when used to route IP traffic (this processing
can result in usage of up to 90%).
• It takes a l ot of memory to store route tables. The memory on the RS/6000
SP node is typically more expensive than router memory.
The CPU on a node can only drive the system I/O bus at less than 80
megabytes per second, which is less than what a high-end router can do.
For these reasons, the performance of routers in handling IP traffic from
remote systems to the RS/6000 SP nodes was limited.
2.1.5 Routing with the GRF
The GRF is a dedicated, high-performance router (see Figure 6). Each SP
Switch Router adapter can route up to 30,000 packets per second and up to
100 MB/s into the SP Switch network in each direction simultaneously.
Node
Node
. . .
SP Switch
Node
GRF
Figure 6. Routing with GRF
12 IBM 9077 SP Switch Router: Get Connec ted to the SP S witch
Internet/Intranet
ATM
FDDI
Ethernet
Loading...