Cisco WS-C4507R, Catalyst 4000, Catalyst 4500 Troubleshooting Manual

CISCO CATALYST 4000 AND 4500 TROUBLESHOOTING

SESSION RST-3508

RST-3508 9805_05_2004_c2

Troubleshooting

• Connectivity

• Performance

• Unexpected feature behavior

Which then impacts connectivity and/or performance

1

RST-3508 9805_05_2004_c2

222

Cisco Catalyst 4500 IOS Supervisor Options

Supervisor V

Optional NetFlow Daughter Card

Supervisor IV

-Based Supervisors

®

Optional NetFlow Daughter Card

Catalyst 4500 Series

Supervisor II-Plus

Cisco IOS

RST-3508 9805_05_2004_c2

• Optimized for Large Networks (Premium HW and SW Services)

• Support for Higher Port Densities (Catalyst 4510R)

• Advanced Layer 3 Switching/Routing (OSPF, EIGRP, IS:IS)

• Highly Scalable Layer 2/3/4 Services

• Supports Up to 10 Active Slots—96Gbps + 72Mpps

• Redundancy Support in 4507R and 4510R Chassis

• Catalyst 4503, 4506, 4507R, 4510R, and 4006 Chassis

• Cisco IOS-Based

• Optimized for Medium Networks

• Advanced Layer 3 Switching/Routing (OSPF, EIGRP, IS:IS)

• Scalable Layer 2/3/4 Services

• Supports Up to 5 Active Slots—64Gbps + 48Mpps

• Redundancy Support in 4507R Chassis

• Advanced Layer 3 Switching

• Catalyst 4503, 4506, 4507R and 4006 Chassis

• Cisco IOS-Based

• Optimized for Smaller Networks

• Basic Layer 3 Switching/Routing (RIP and Static)

• Layer 2/3/4 Intelligent Services

• Supports Up to 5 Active Slots—64Gbps + 48Mpps

• Redundancy Support in 4507R Chassis

• Catalyst 4503, 4506, 4507R and 4006 Chassis

• Cisco IOS-Based

Catalyst 4500 Supervisor Engines Key Differences Among II+, IV and V

333

Basic L2/3/4 ServicesMulti-Layer Switching

RST-3508 9805_05_2004_c2

Supervisor IVSupervisor II-Plus

Full L2/3/4 Services and

Routing

No NoMulticast Suppression

Non-Blocking Gig OnlyNon-Blocking Gig OnlyQoS Sharing

Pass-ThroughPass-ThroughQinQ

22Active Redundant Sup Uplinks

Supervisor V

96 Gbps64 Gbps64 GbpsSwitching Capacity

72 Mpps48 Mpps48 MppsThroughput

Full L2/3/4 Services and Routing

YesYesNo(E)IGRP,OSPF,BGP, ISIS

YesYesYesRIP, Static Routes

C4006, C4503, C4505, C4507, C4510C4006, C4503, C4505, C4507C4006,C4503,C4505,C4507Chassis Support

400 MHz333 MHz266 MHzCPU

128K128K32KIP CEF Entries

512512256SDRAM

4K4K2KActive VLANs

28K(L3) 16K (L2)28K(L3) 16K (L2)12K(L3) 16K (L2)Multicast Entries

3K3K1.5KSTP Instance

4K4K1KSVI

Yes (512KB)Yes (512KB)Flash Simulated NVRAMNVRAM

Yes (16K)Yes (16K)Yes (8K)IGMP Snooping

YesYesNoNetflow Support

HardwareSoftwareSoftwareBroadcast Suppression

Yes

All Ports

In Hardware

4

444

Catalyst 4500 Series: Cisco IOS Software Options

• Single Cisco IOS image across all switches

• Basic (cat4000-i9s-mz):

RIP v1/2, static routes, AppleTalk, IPX

• Enhanced (cat4000-i5s-mz)

(Supervisor engines IV, V): OSPF, (E)IGRP, BGP, IS-IS

• Crypto images basic (cat4000-i9k91s-mz ) and

enhanced (cat4000-i5k91s-mz) provide :

SSH v1 SSH v2 (12.1.19EW and higher)

• Multicast, PBR, security is included in all

images

• Redundancy is supported for all images

• Supervisor II-Plus supports only the basic

images

RST-3508 9805_05_2004_c2

Cisco IOS Versions for Cisco IOS-Based Supervisors

• The GD train 12.1.20E is based on the

features in Cisco IOS 12.1.(12c) EW

• The Cisco IOS 12.2(18) EW release will be

the ongoing maintenance release vehicle

• For the latest features always use latest

CCO EW release

555

RST-3508 9805_05_2004_c2

666

Show Version (Sup II-Cat OS)

RST-3508 9805_05_2004_c2

cat4503> (enable) show version

WS-C4503 Software, Version NmpSW: 7.4(1)

NMP S/W compiled on Sep 20 2002, 11:46:26

GSP S/W compiled on Sep 20 2002, 11:24:50

System Bootstrap Version: 5.4(1)

Hardware Version: 2.0 Model: WS-C4503 Serial #: FOX07071SXT

Mod Port Model Serial # Versions

--- ---- ------------------ -------------------- -------------------------------

--

1 2 WS-X4013 JAB0437072X Hw : 2.0

2 48 WS-X4148 JAB034401CJ Hw : 1.6

DRAM FLASH NVRAM

Module Total Used Free Total Used Free Total Used Free

------ ------- ------- ------- ------- ------- ------- ----- ----- -----

1 65536K 39128K 26408K 16384K 10058K 6326K 480K 302K 178K

Uptime is 20 days, 14 hours, 45 minutes

Minimum

Cat OS for

4500

Chassis

Gsp: 7.4(1.0)

Nmp: 7.4(1)

777

Show Version (Cisco IOS Supervisors)

cat4500#sh version

Cisco Internetwork Operating System Software

IOS (tm) Catalyst 4000 L3 Switch Software (cat4000-I5S-M), Version 12.2(18)EW, EARLY DEPLOYMENT RELEASE

SOFTWARE (fc1)

TAC Support: http://www.cisco.com/tac

Compiled Fri 20-Dec-02 13:52 by eaarmas

Image text-base: 0x00000000, data-base: 0x00E638AC

Compiled Fri 30-Jan-04 01:55 by hqluong

Image text-base: 0x00000000, data-base: 0x010B0624

ROM: 12.1(12r)EW

Dagobah Revision 90, Swamp Revision 24

r3_4507R_S4 uptime is 3 weeks, 6 days, 18 hours, 39 minutes

System returned to ROM by reload

System restarted at 17:00:36 PST Wed Mar 24 2004

System image file is "bootflash:cat4000-i5s-mz.122-18.EW.bin“

cisco WS-C4507R (XPC8245) processor (revision 4) with 524288K bytes of memory.

Processor board ID FOX062105FU

Last reset from Redundancy Reset

76 Gigabit Ethernet/IEEE 802.3 interface(s)

403K bytes of non-volatile configuration memory.

Configuration register is 0x2102

RST-3508 9805_05_2004_c2

Minimum

IOS is

12.1(12c)EW for 4500 Chassis

888

Disaster Recovery: Using Management Port, fa1, for Network Boot

rommon 2 >set

rommon 5 >set interface fa1 172.20.64.158 255.255.255.0

rommon 3 >set ip route def ault 172.20.64.1

rommon 7 >set TftpServer 1 72.20.64.136

rommon 6 >ping 172.20.64.1 36

Host 172.20.64.136 is alive

rommon 11 >boot tftp://172.20.64.136/tftpboot/cat4000-i5s-mz.122-18.EW.bin

Tftp Session details are ....

Filename : /tftpboot/cat 4000-i5s-mz.122-18.EW.bin IP Address : 172.20.64.158 Loading from TftpServer: 172 .20.64.136

Received data packet # 20019

Loaded 10249540 bytes successfully .

RST-3508 9805_05_2004_c2

use the set command w no options to check for and verify IP settings

999

Agenda

• Redundancy

• Hardware and Related Issues

• Unicast Packet Forwarding

• Multicast Packet Forwarding

• ACLs

• QoS

RST-3508 9805_05_2004_c2

101010

REDUNDANCY

RST-3508 9805_05_2004_c2

Supervisor Redundancy (4507R/4510R)

• Route Processor Redundancy (RPR)

• One supervisor active

• Other supervisor suspended during bootup

Console to standby supervisor not available thereafter

Cisco Internetwork Operating System Software IOS (tm) Catalyst 4000 L3 Switch Software (cat4000-I5S-M), Version 12.2(18)EW, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1) TAC Support: http://www.cisco.com/tac Copyright (c) 1986-2004 by cisco Systems, Inc. Compiled Fri 30-Jan-04 01:55 by hqluong Image text-base: 0x00000000, data-base: 0x010B0624

*********************************** * STANDBY SUPERVISOR * * REDUNDANCY mode is RPR * * Waiting for Switchover Activity * ***********************************

111111

RST-3508 9805_05_2004_c2

121212

Supervisor Redundancy

During Switchover the Standby Supervisor

• Completes the booting sequence

*********************************** * STANDBY SUPERVISOR * * REDUNDANCY mode is RPR * * Waiting for Switchover Activity * *********************************** cisco WS-C4507R (MPC8245) processor (revision 4) with 524288K bytes of memory. Processor board ID FOX062105G1 Last reset from Reload 1 Virtual Ethernet/IEEE 802.3 interface(s) 96 FastEthernet/IEEE 802.3 interface(s) 26 Gigabit Ethernet/IEEE 802.3 interface(s) 403K bytes of non-volatile configuration memory.

Uncompressed configuration from 7028 bytes to 17442 bytes

• Resets the modules so they can perform self diagnostics

• Parses the configuration

• Waits for the modules to come online and links to get

established

• Builds routing tables, MAC-address tables, and other dynamic

protocols

RST-3508 9805_05_2004_c2

131313

Cisco Catalyst 4507R/4510R Supervisor Redundancy

What Is Synchronized?

• Startup configuration

(by issuing the write memory command)

• Boot-variable

• Configuration-register

• Calendar

• VLAN database

RST-3508 9805_05_2004_c2

141414

Cisco Catalyst 4507R/4510R Supervisor Redundancy

What Is Not Synchronized?

• Running configurations

• Routing table/FIB/adjacency table

• MAC-address table

• Cisco IOS images: Should be the same

(not enforced by software)

RST-3508 9805_05_2004_c2

Accessing the Standby Supervisor

Console Port Is Not Available After Initialization State

Commands

dir slavebootflash:

dir slaveslot0:

del slave bootflash: < filename>

del slaveslot0: < filename>

squeeze slavebootflash:

squeeze slaveslot0:

format slavebootflash:

format slaveslot0:

copy <source> slavebootflash:

copy <source>slaveslot0:

Performs Squeeze Function After

Delete to Recover Device Space

Supervisor Files or TFTP Server

Description

Lists Contents

Deletes Specific Files

Format the Standby

Source Could Be Active

151515

RST-3508 9805_05_2004_c2

161616

Supervisor Redundancy

cat4507R#sh module

Chassis Type : WS-C4507R

Power consumed by backplane : 40 Watts

Mod Ports Card Type Model Serial No.

---+-----+--------------------------------------+------------------+-----------

1 2 1000BaseX (GBIC) Supervisor(active) WS-X4013+ JAB071904FP

2 2 1000BaseX (GBIC) Supervisor(standby) WS-X4013+ JAB071904FD

3 48 10/100BaseTX (RJ45)V, Cisco/IEEE WS-X4248-RJ45V JAB074005BE

M MAC addresses Hw Fw Sw Status

--+--------------------------------+---+------------+----------------+---------

1 0009.e845.6300 to 0009.e845.6301 0.3 12.1(20r)EW 12.2(18)EW, EARL Ok

2 0009.e845.6302 to 0009.e845.6303 0.3 12.1(19r)EW 12.2(18)EW, Ok

3 0005.9a80.3c00 to 0005.9a80.3c2f 0.9 Ok

System Failures:

----------------

Power Supply: bad/off (see 'show power')

RST-3508 9805_05_2004_c2

171717

Supervisor Redundancy

cat4507R#sh mod

Chassis Type : WS-C4507R

Power consumed by backplan e : 40 Watts

Mod Ports Card Type Model Serial No.

---+-----+---------------- ----------------------+------------------+-----------

1 2 1000BaseX (GBIC) Su pervisor(active) WS-X4013+ JAB071904FP

2 Standby Supervisor

3 48 10/100BaseTX (RJ45)V, Cisco/IEEE WS-X4248-RJ45V JAB074005BE

M MAC addresses Hw Fw Sw Status

--+----------------------- ---------+---+------------+----------------+---------

1 0009.e845.6300 to 0009.e845.6301 0.3 12.1(20r)EW 12.2(18)EW, EARL Ok

2 Unknown Unknown Unknown Other

3 0005.9a80.3c00 to 0005.9a80.3c2f 0.9 Ok

System Failures:

----------------

Power Supply: bad/off (see 'show power')

RST-3508 9805_05_2004_c2

181818

Power Supply Redundancy

cat4500(config)#power redundancy-mode ?

combined combine power supply outputs (no redundancy)

redundant either power supply can operate system (redundancy)

RST-3508 9805_05_2004_c2

Power Supply Redundancy

cat4507R#show power

Power Fan Inline

Supply Model No Type Status Sensor Status

------ ---------------- --------- ----------- ------- -------

PS1 PWR-C45-2800AC AC 2800W bad/off bad/off bad/off

PS2 PWR-C45-1400AC AC 1400W good good n.a.

*** Power Supplies of different type have been detected***

Power supplies needed by system : 2

Power supplies currently available : 1

*** Insufficient power supplies present for specified configuration ***

Power Summary Maximum

(in Watts) Used Available

---------------------- ---- ---------

System Power (12V) 335 1360

Inline Power (-50V) 0 0

Backplane Power (3.3V) 40 40

---------------------- ----

Total Used 375 (not to exceed Total Maximum Available = 1400)

RST-3508 9805_05_2004_c2

191919

202020

HSRP Redundancy

• HSRP configured between supervisors on

different Catalyst 4500 chassis

• HSRP configured between a Catalyst 4500

and an external router

• No HSRP between two supervisors on the

same chassis since the standby supervisor is in suspended mode

RST-3508 9805_05_2004_c2

Port Channel Redundancy

• Layer 2 EtherChannel

All interfaces in the same VLAN or trunks with trunking mode the same on both ends

When trunking…allowed ranges of VLANs must be the same

• Layer 3 EtherChannel bundles

Port channel must be created first; then...

Use “no switchport” to create the layer 3 ports in the channel

• Supervisor Engine ports in an EtherChannel

Supervisor II: Both ports can be in the EtherChannel

Single Cisco IOS supervisor in slot 1 or 2: gi1/1–2 or gi2/1–2 active

With Sup II-Plus or IV Dual Supervisors, only gig1/1 AND gig2/1 active; but gig1/2 and gig2/2 can be placed in an EtherChannel bundle for backup

With Supervisor V, all four uplinks are active

®

bundles

212121

RST-3508 9805_05_2004_c2

222222

Channel Troubleshooting

Commands Similar to the PAgP Commands Are Available for LACP

r3_4506#sh etherchannel summary(truncated output) Flags: D - down P - in port-channel

I - stand-alone s - suspended

R - Layer3 S - Layer2

U - in use

Number of channel-groups in use: 2 Number of aggregators: 2

Group Port-channel Protocol Ports

------+-------------+-----------+--------------------------1 Po1(SU) PAgP Gi1/1(P) Gi1/2(P) 2 Po2(RU) PAgP Fa3/46(P) Fa3/47(P)

r3_4507R_S4#sh int gig1/1 etherchannel (truncated) Port state = Up Cnt-bndl Suspend Not-in-Bndl Flags: S - Device is sending Slow hello. C - Device is in Consistent state.

A - Device is in Auto mode. P - Device learns on physical port. d - PAgP is down.

Local information:

Port Flags State Timers Interval Count Priority Method Ifindex Gi1/1 d U1/S1 1s 0 128 Any 0

Age of the port in the current state: 27d:19h:18m:59s

Probable reason: Source monitor interfaces are not allowed to be part of an etherchannel

Hello Partner PAgP Learning Group

Ages Every 30 Secs

RST-3508 9805_05_2004_c2

r3_4506 #sh int port-channel 1 ( truncated)

Port-channel1 is up, line protocol is up (connected)

Description: to cat4507R

MTU 1500 bytes, BW 2000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex, 1000Mb/s, media type is N/A

Members in this channel: Gi1/1 Gi1/2

r3_4506(config)#port-channel load-balance ?

r3_4506sh etherchannel load-balance Source XOR Destination IP address

r3_4506#sh pagp neighbor Flags: S - Device is sending Slow hello. C - Device is in Consistent state.

A - Device is in Auto mode. P - Device learns on physical port.

Channel group 1 neighbors

Partner Partner Partner Partner Group Port Name Device ID Port Age Flags Cap. Gi1/1 r3_4507R_S4.cisco.co 0009.e845.5f00 Gi1/1 17s SC 10001 Gi1/2 r3_4507R_S4.cisco.co 0009.e845.5f00 Gi2/1 24s SC 10001

Global Command

dst-ip Dst IP Addr dst-mac Dst Mac Addr dst-port Dst TCP/UDP Port src-dst-ip Src XOR Dst IP Addr src-dst-mac Src XOR Dst Mac Addr src-dst-port Src XOR Dst TCP/UDP Port src-ip Src IP Addr src-mac Src Mac Addr src-port Src TCP/UDP Port

232323

HARDWARE AND RELATED ISSUES

RST-3508 9805_05_2004_c2

242424

Switch Management Interfaces Cisco Catalyst OS

Cat4K-c (enable) sh int

sl0: flags=50<DOWN,POINTOPOINT,RUNNING>

slip 0.0.0.0 dest 0.0.0.0

sc0: flags=63<UP,BROADCAST,RUNNING>

vlan 1 inet 1.1.1.3 netmask 255.255.255.0 broadcast 1.1.1.255

me1: flags=62<DOWN,BROADCAST,RUNNING>

inet 0.0.0.0 netmask 0.0.0.0 broadcast 0.0.0.0

• sc0 inband management interface

• sc0 connects to switching fabric

• sc0 participates in STP, CDP, VLAN membership

• sl0 and me1 out-of-band management interfaces

• sl0 and me1 do not connect to switching fabric

• sl0 and me1 do not participate in STP, CDP, VLAN membership

• Only one out of sc0 and me1 can be up

RST-3508 9805_05_2004_c2

252525

Switch Ports/Interfaces

• On Cisco Catalyst OS switches these are

Layer 2 ports

• On Cisco IOS switches these can be

Layer 3 routed interfaces

Layer 3 Switched Virtual interfaces (SVIs)

Layer 3 portchannel interfaces

Layer 2 switchport interfaces—access or trunk

Layer 2 portchannel interfaces

• By default on Cisco IOS switches the interfaces are

Layer 2 switchport interfaces

• “no switchport” command converts these to Layer

3 routed interfaces

RST-3508 9805_05_2004_c2

262626

High CPU Usage-Supervisor II

Console> (enable) show proc cpu

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

--- ----------- ---------- -------- ------- ------- ------- --- ---------------

(truncated)

98 23438905 7904296 9352 86.64% 89.57% 87.50% 0 Switching overhead 99 2271479 1443242 57968 1.19% 1.04% 0.98% 0 Admin overhead

(truncated)

Remember: Kernel and Idle CPU Usage Is the Percentage of Time the CPU Was Idle

Console> (enable) sh proc cpu

(truncated)

CPU utilization for five seconds: 14.45%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

--- ----------- ---------- -------- ------- ------- ------- --- --------------1 345976604 0 0 85.55% 85.00% 85.00% -2 Kernel and Idle

RST-3508 9805_05_2004_c2

one minute: 15.00%

five minutes: 15.00%

272727

High CPU Usage-Supervisor II

• Switching overhead

Address learning (path setup) for new MAC addresses

Normal host entry aging, as well as fast aging due to reception of STP topology Change notification

Packet processing for control traffic such as STP BPDUs, CDP, VTP, DTP, PAgP, and so forth

Packet processing for management traffic such as telnet, SNMP, and HTTP

• Admin overhead

Switch fabric Application Specific Integrated Circuit (ASIC) and other hardware management

Line card ASIC management

Port monitoring

RST-3508 9805_05_2004_c2

282828

Cisco IOS Supervisor CPU Usage

Total CPU Utilization (Process + Interrupt)

cat4500# sh proc cpu CPU utilization for five seconds: 73%/17%; one minute: 74%; five minutes: 76% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

1 2358396 1705816 1382 0.32% 1.17% 0.68% 0 IP-EIGRP Router 2 2337276 21210 110196 0.00% 0.80% 0.89% 0 Check heaps 8 6551276 3786002 1730 3.05% 4.41% 4.70% 0 IP Input

9 24211844 1644250 14725 27.91% 26.06% 25.45% 0 22 15663744 474459 33014 19.71% 20.67% 21.89% 0 TCP Driver 32 508 36 14111 5.07% 0.73% 0.15% 13 Virtual Exec

Interrupt level

Average

Interrupt level

Processing Time

• Make sure to distinguish interrupt and process level

• A CPU utilization value of 20% to 50% is normal, even under

minimal load with Power over Ethernet (PoE) line cards

RST-3508 9805_05_2004_c2

TCP Timer

292929

CPU Troubleshooting Commands: Cisco IOS Supervisor

cat4500# sh proc cpu CPU utilization for five seconds: 99%/0%; one minute: 27%; five minutes: 15%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

(truncated)

25 1599072 5303348 301 10.01% 9.20% 8.20% 0 Cat4k Mgmt HiPri 26 1869444 522959 3574 86.16% 78.67% 36.07% 0 Cat4k Mgmt LoPri

• The Mgt Hi-Priority and Mgt Low-Priority are two processes

that Cisco Catalyst 4500 platform code runs

• These two processes share total CPU usage among other

Cisco IOS processes

The reason there are two processes is when a job in Cisco Catalyst 4500 platform takes longer time than expected or exceeds the max allocated CPU time, the job would be run under low-priority for sometime until it's usage is reduced, this would give other higher priority (IOS) process a chance to run

• If the above two processes consume most of the CPU time

further investigation is now needed

RST-3508 9805_05_2004_c2

303030

High CPU Usage: Cisco IOS Supervisors

• CPU usage is not indicative of forwarding

performance on any supervisor

• Forwarding decisions are made in hardware

• Packets sent to the CPU

Control packets: STP, CDP, PAgP, LACP, UDLD Routing protocol updates IPX/Appletalk SNMP/telnet/ ARP responses to ARP requests Packets with IP options/expired TTL or non-ARPA encaps Packets with special handling, i.e. tunneling, encryption ACL logging enabled Input ACL drops MTU check failure Adjacency same interface

RST-3508 9805_05_2004_c2

313131

Extras: QoS on the CPU Port

0: ESMP

1: Control

Packets to the CPU

…

CPU Queues

• Protects important traffic when CPU usage is high

BPDUs/routing updates get priority

RST-3508 9805_05_2004_c2

Can still telnet or SNMP query when CPU is high

2: Host Learning 3–5: L3 Forwarding

9–10: L3 Rx (Telnet/SNMP)

15: MTU Fail/Invalid

323232

Show Platform CPU Statistics Fields:

• ESMP:

Even Simpler Management Protocol…used by the CPU for reading line card status: link, speed, led, etc.

• Control:

L2 control plane packets go here—STP, CDP, PaGP, LACP, UDLD, etc.

• Host learning:

Packets with unknown L2 source address are copied to CPU to build CAM table

• L3 fwd:

GRE tunnels Gleaning

• L2 fwd: Any non-IP switchable packet

IPX/Appletalk Zero TTL field Non-ARPA encapsulated packets ARP’s

• L3 Rx:

L3 packets to the switch—SNMP, telnet, ping

• ACL forward

RST-3508 9805_05_2004_c2

EIGRP/OSPF updates

333333

CPU Troubleshooting Commands: Cisco IOS Supervisor

cat4500# show platform cpu packet statistics (all)>>> lots of output, look for Total packet queues 16

Packets Received by Packet Queue

Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg

---------------------- --------------- --------- --------- --------- ---------Esmp 42808 38 38 34 6 Control 9919 11 10 8 1 Host Learning 39 0 0 0 0 L3 Fwd High 0 0 0 0 0 L3 Fwd Medium 0 0 0 0 0 L3 Fwd Low 0 0 0 0 0 L2 Fwd High 0 0 0 0 0 L2 Fwd Medium 0 0 0 0 0 L2 Fwd Low 99929 0 5 92 17 L3 Rx High 0 0 0 0 0 L3 Rx Low 36 0 0 0 0 RPF Failure 0 0 0 0 0 ACL fwd(snooping) 1165 1 1 1 0 ACL log, unreach 0 0 0 0 0 ACL sw processing 0 0 0 0 0 MTU Fail/Invalid 0 0 0 0 0

L2 and L3 High, Medium, and Low Are Based on the DSCP/COS Field of the Packet

RST-3508 9805_05_2004_c2

343434

CPU Troubleshooting Commands: Cisco IOS Supervisor

Look for Received Packets and Rx Drops

cat4500# sh platform cpu packet driver ( truncated)

Queue rxTail received all guar allJ gurJ rxDrops rxDelays 0 Esmp 63A6B70 25708 100 100 0 5 0 0 1 Control 63A6CF4 5405 595 600 0 5 0 0 2 Host Learning 63A76A0 24 500 500 0 5 0 0 3 L3 Fwd High 63A7E10 0 300 300 0 5 0 0 4 L3 Fwd Medium 63A82C0 0 500 500 0 5 0 0 5 L3 Fwd Low 63A8A90 0 900 900 0 5 0 0 6 L2 Fwd High 63A98A0 0 300 300 0 5 0 0 7 L2 Fwd Medium 63A9D50 0 500 500 0 5 0 0

8 L2 Fwd Low 63AB2E4 99929 899 900 0 5 434063 0

RST-3508 9805_05_2004_c2

Monitor the CPU on Cisco Catalyst 4500

• Incoming packets are allowed on SPAN destination port

• Monitor the CPU port (excellent in capturing high CPU util)

• Cisco Catalyst 4500: Only LAN Switch where you can monitor on the CPU port

MAC

Table

FFE

CPU

Source Port/

VLAN

RST-3508 9805_05_2004_c2

Monitor

Destination

Port

Modules

Input

Packet

Enable

Inpkts

Switch Fabric

PPE

Packet

Memory

Mirror Source Ports, VLANs, CPU With

SPAN Capabilities on Catalyst 4500

ACL

TCAM

353535

363636

Monitoring the CPU on Cisco Catalyst 4500 IOS Supervisors

r3_4506 #sh monitor session 1 Session 1

--------Type : Local Session Source Ports :

RX Only : CPU(Queues: 32)

Destination Ports : Fa3/37

Encapsulation : Native

Ingress : Disabled

Learning : Disabled

RST-3508 9805_05_2004_c2

C(config)#monitor session 1 source cpu queue ?

<1-32> SPAN source CPU queue numbers

acl Input and output ACL [13-20]

adj-same-if Packets routed to the incoming interface

[7]

all All queues [1-32]

bridged L2/bridged packets [29-32]

control-packet Layer 2 Control Packets [5]

mtu-exceeded Output interface MTU exceeded [9]

nfl Packets sent to CPU by netflow (unused) [8]

routed L3/routed packets [21-28]

rpf-failure Multicast RPF Failures [6]

span SPAN to CPU (unused) [11]

unknown-sa Packets with missing source address [10]

373737

Ping Latency

• Low priority task on the CPU

• Response times of 7–10 ms are typical on

an idle switch

• Pings through the switch are handled as

ordinary data packets and switched in HW

RST-3508 9805_05_2004_c2

383838

Best Practices

• Baseline the CPU in steady state

Normally which processes are causing the highest CPU usage

• When troubleshooting

Are high CPU processes different from the baseline?

Is the CPU consistently elevated or just spiking?

Are there TCNs in the network caused by flapping ports?

Is there excessive broadcast or multicast traffic in the management subnet or VLAN?

Is there excessive management traffic such as SNMP polling?

• Isolate the management VLAN from VLANs with

user data traffic

Particularly heavy broadcast traffic such as IPX or AppleTalk

RST-3508 9805_05_2004_c2

Cisco Catalyst 4500 Architecture

• 3-slot, 6-slot chassis—one supervisor

with two or five module slots

• 7-slot chassis—one or two

supervisors with five module slots

• 10-slot chassis—one or two

supervisors (Supervisor V only) with eight module slots

• Cisco IOS supervisors provide:

Central forwarding engine (Fast Forwarding Engine, FFE)

Buffering and 64 Gbps fabric (Packet Processing Engine, PPE)…96 Gbps fabric with Supervisor V and PPE2

• 12 Gbps bandwidth to each module

• Auto MDIX on 10/100/1000 Ports

• Modules are transparent:

Contain simple “stub” ASICs, PHYs

RST-3508 9805_05_2004_c2

No buffering or local switching

Switching

Module

Note: Supervisor Engine V Supports 3 Additional Line Card Slots

Forwarding Engine (FFE)

Shared Memory

Fabric (PPE)

12 Gbps

Switching

Module

12 Gbps

Switching

Module

12 Gbps

Switching

Module

12 Gbps

393939

12 Gbps

Switching

Module

404040

Cisco WS-C4507R, Catalyst 4000, Catalyst 4500 Troubleshooting Manual

Specifications and Main Features

Frequently Asked Questions

User Manual