Cisco WS-C4507R, Catalyst 4000, Catalyst 4500 Troubleshooting Manual

CISCO CATALYST 4000 AND 4500 TROUBLESHOOTING

SESSION RST-3508

RST-3508 9805_05_2004_c2

Troubleshooting

• Connectivity

• Performance

• Unexpected feature behavior

Which then impacts connectivity and/or performance

1

RST-3508 9805_05_2004_c2

222

Cisco Catalyst 4500 IOS Supervisor Options

Supervisor V

Optional NetFlow Daughter Card

Supervisor IV

-Based Supervisors

®

Optional NetFlow Daughter Card

Catalyst 4500 Series

Supervisor II-Plus

Cisco IOS

RST-3508 9805_05_2004_c2

• Optimized for Large Networks (Premium HW and SW Services)

• Support for Higher Port Densities (Catalyst 4510R)

• Advanced Layer 3 Switching/Routing (OSPF, EIGRP, IS:IS)

• Highly Scalable Layer 2/3/4 Services

• Supports Up to 10 Active Slots—96Gbps + 72Mpps

• Redundancy Support in 4507R and 4510R Chassis

• Catalyst 4503, 4506, 4507R, 4510R, and 4006 Chassis

• Cisco IOS-Based

• Optimized for Medium Networks

• Advanced Layer 3 Switching/Routing (OSPF, EIGRP, IS:IS)

• Scalable Layer 2/3/4 Services

• Supports Up to 5 Active Slots—64Gbps + 48Mpps

• Redundancy Support in 4507R Chassis

• Advanced Layer 3 Switching

• Catalyst 4503, 4506, 4507R and 4006 Chassis

• Cisco IOS-Based

• Optimized for Smaller Networks

• Basic Layer 3 Switching/Routing (RIP and Static)

• Layer 2/3/4 Intelligent Services

• Supports Up to 5 Active Slots—64Gbps + 48Mpps

• Redundancy Support in 4507R Chassis

• Catalyst 4503, 4506, 4507R and 4006 Chassis

• Cisco IOS-Based

Catalyst 4500 Supervisor Engines Key Differences Among II+, IV and V

333

Basic L2/3/4 ServicesMulti-Layer Switching

RST-3508 9805_05_2004_c2

Supervisor IVSupervisor II-Plus

Full L2/3/4 Services and

Routing

No NoMulticast Suppression

Non-Blocking Gig OnlyNon-Blocking Gig OnlyQoS Sharing

Pass-ThroughPass-ThroughQinQ

22Active Redundant Sup Uplinks

Supervisor V

96 Gbps64 Gbps64 GbpsSwitching Capacity

72 Mpps48 Mpps48 MppsThroughput

Full L2/3/4 Services and Routing

YesYesNo(E)IGRP,OSPF,BGP, ISIS

YesYesYesRIP, Static Routes

C4006, C4503, C4505, C4507, C4510C4006, C4503, C4505, C4507C4006,C4503,C4505,C4507Chassis Support

400 MHz333 MHz266 MHzCPU

128K128K32KIP CEF Entries

512512256SDRAM

4K4K2KActive VLANs

28K(L3) 16K (L2)28K(L3) 16K (L2)12K(L3) 16K (L2)Multicast Entries

3K3K1.5KSTP Instance

4K4K1KSVI

Yes (512KB)Yes (512KB)Flash Simulated NVRAMNVRAM

Yes (16K)Yes (16K)Yes (8K)IGMP Snooping

YesYesNoNetflow Support

HardwareSoftwareSoftwareBroadcast Suppression

Yes

All Ports

In Hardware

4

444

Catalyst 4500 Series: Cisco IOS Software Options

• Single Cisco IOS image across all switches

• Basic (cat4000-i9s-mz):

RIP v1/2, static routes, AppleTalk, IPX

• Enhanced (cat4000-i5s-mz)

(Supervisor engines IV, V): OSPF, (E)IGRP, BGP, IS-IS

• Crypto images basic (cat4000-i9k91s-mz ) and

enhanced (cat4000-i5k91s-mz) provide :

SSH v1 SSH v2 (12.1.19EW and higher)

• Multicast, PBR, security is included in all

images

• Redundancy is supported for all images

• Supervisor II-Plus supports only the basic

images

RST-3508 9805_05_2004_c2

Cisco IOS Versions for Cisco IOS-Based Supervisors

• The GD train 12.1.20E is based on the

features in Cisco IOS 12.1.(12c) EW

• The Cisco IOS 12.2(18) EW release will be

the ongoing maintenance release vehicle

• For the latest features always use latest

CCO EW release

555

RST-3508 9805_05_2004_c2

666

Show Version (Sup II-Cat OS)

RST-3508 9805_05_2004_c2

cat4503> (enable) show version

WS-C4503 Software, Version NmpSW: 7.4(1)

NMP S/W compiled on Sep 20 2002, 11:46:26

GSP S/W compiled on Sep 20 2002, 11:24:50

System Bootstrap Version: 5.4(1)

Hardware Version: 2.0 Model: WS-C4503 Serial #: FOX07071SXT

Mod Port Model Serial # Versions

--- ---- ------------------ -------------------- -------------------------------

--

1 2 WS-X4013 JAB0437072X Hw : 2.0

2 48 WS-X4148 JAB034401CJ Hw : 1.6

DRAM FLASH NVRAM

Module Total Used Free Total Used Free Total Used Free

------ ------- ------- ------- ------- ------- ------- ----- ----- -----

1 65536K 39128K 26408K 16384K 10058K 6326K 480K 302K 178K

Uptime is 20 days, 14 hours, 45 minutes

Minimum

Cat OS for

4500

Chassis

Gsp: 7.4(1.0)

Nmp: 7.4(1)

777

Show Version (Cisco IOS Supervisors)

cat4500#sh version

Cisco Internetwork Operating System Software

IOS (tm) Catalyst 4000 L3 Switch Software (cat4000-I5S-M), Version 12.2(18)EW, EARLY DEPLOYMENT RELEASE

SOFTWARE (fc1)

TAC Support: http://www.cisco.com/tac

Compiled Fri 20-Dec-02 13:52 by eaarmas

Image text-base: 0x00000000, data-base: 0x00E638AC

Compiled Fri 30-Jan-04 01:55 by hqluong

Image text-base: 0x00000000, data-base: 0x010B0624

ROM: 12.1(12r)EW

Dagobah Revision 90, Swamp Revision 24

r3_4507R_S4 uptime is 3 weeks, 6 days, 18 hours, 39 minutes

System returned to ROM by reload

System restarted at 17:00:36 PST Wed Mar 24 2004

System image file is "bootflash:cat4000-i5s-mz.122-18.EW.bin“

cisco WS-C4507R (XPC8245) processor (revision 4) with 524288K bytes of memory.

Processor board ID FOX062105FU

Last reset from Redundancy Reset

76 Gigabit Ethernet/IEEE 802.3 interface(s)

403K bytes of non-volatile configuration memory.

Configuration register is 0x2102

RST-3508 9805_05_2004_c2

Minimum

IOS is

12.1(12c)EW for 4500 Chassis

888

Disaster Recovery: Using Management Port, fa1, for Network Boot

rommon 2 >set

rommon 5 >set interface fa1 172.20.64.158 255.255.255.0

rommon 3 >set ip route def ault 172.20.64.1

rommon 7 >set TftpServer 1 72.20.64.136

rommon 6 >ping 172.20.64.1 36

Host 172.20.64.136 is alive

rommon 11 >boot tftp://172.20.64.136/tftpboot/cat4000-i5s-mz.122-18.EW.bin

Tftp Session details are ....

Filename : /tftpboot/cat 4000-i5s-mz.122-18.EW.bin IP Address : 172.20.64.158 Loading from TftpServer: 172 .20.64.136

Received data packet # 20019

Loaded 10249540 bytes successfully .

RST-3508 9805_05_2004_c2

use the set command w no options to check for and verify IP settings

999

Agenda

• Redundancy

• Hardware and Related Issues

• Unicast Packet Forwarding

• Multicast Packet Forwarding

• ACLs

• QoS

RST-3508 9805_05_2004_c2

101010

REDUNDANCY

RST-3508 9805_05_2004_c2

Supervisor Redundancy (4507R/4510R)

• Route Processor Redundancy (RPR)

• One supervisor active

• Other supervisor suspended during bootup

Console to standby supervisor not available thereafter

Cisco Internetwork Operating System Software IOS (tm) Catalyst 4000 L3 Switch Software (cat4000-I5S-M), Version 12.2(18)EW, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1) TAC Support: http://www.cisco.com/tac Copyright (c) 1986-2004 by cisco Systems, Inc. Compiled Fri 30-Jan-04 01:55 by hqluong Image text-base: 0x00000000, data-base: 0x010B0624

*********************************** * STANDBY SUPERVISOR * * REDUNDANCY mode is RPR * * Waiting for Switchover Activity * ***********************************

111111

RST-3508 9805_05_2004_c2

121212

Supervisor Redundancy

During Switchover the Standby Supervisor

• Completes the booting sequence

*********************************** * STANDBY SUPERVISOR * * REDUNDANCY mode is RPR * * Waiting for Switchover Activity * *********************************** cisco WS-C4507R (MPC8245) processor (revision 4) with 524288K bytes of memory. Processor board ID FOX062105G1 Last reset from Reload 1 Virtual Ethernet/IEEE 802.3 interface(s) 96 FastEthernet/IEEE 802.3 interface(s) 26 Gigabit Ethernet/IEEE 802.3 interface(s) 403K bytes of non-volatile configuration memory.

Uncompressed configuration from 7028 bytes to 17442 bytes

• Resets the modules so they can perform self diagnostics

• Parses the configuration

• Waits for the modules to come online and links to get

established

• Builds routing tables, MAC-address tables, and other dynamic

protocols

RST-3508 9805_05_2004_c2

131313

Cisco Catalyst 4507R/4510R Supervisor Redundancy

What Is Synchronized?

• Startup configuration

(by issuing the write memory command)

• Boot-variable

• Configuration-register

• Calendar

• VLAN database

RST-3508 9805_05_2004_c2

141414

Cisco Catalyst 4507R/4510R Supervisor Redundancy

What Is Not Synchronized?

• Running configurations

• Routing table/FIB/adjacency table

• MAC-address table

• Cisco IOS images: Should be the same

(not enforced by software)

RST-3508 9805_05_2004_c2

Accessing the Standby Supervisor

Console Port Is Not Available After Initialization State

Commands

dir slavebootflash:

dir slaveslot0:

del slave bootflash: < filename>

del slaveslot0: < filename>

squeeze slavebootflash:

squeeze slaveslot0:

format slavebootflash:

format slaveslot0:

copy <source> slavebootflash:

copy <source>slaveslot0:

Performs Squeeze Function After

Delete to Recover Device Space

Supervisor Files or TFTP Server

Description

Lists Contents

Deletes Specific Files

Format the Standby

Source Could Be Active

151515

RST-3508 9805_05_2004_c2

161616

Supervisor Redundancy

cat4507R#sh module

Chassis Type : WS-C4507R

Power consumed by backplane : 40 Watts

Mod Ports Card Type Model Serial No.

---+-----+--------------------------------------+------------------+-----------

1 2 1000BaseX (GBIC) Supervisor(active) WS-X4013+ JAB071904FP

2 2 1000BaseX (GBIC) Supervisor(standby) WS-X4013+ JAB071904FD

3 48 10/100BaseTX (RJ45)V, Cisco/IEEE WS-X4248-RJ45V JAB074005BE

M MAC addresses Hw Fw Sw Status

--+--------------------------------+---+------------+----------------+---------

1 0009.e845.6300 to 0009.e845.6301 0.3 12.1(20r)EW 12.2(18)EW, EARL Ok

2 0009.e845.6302 to 0009.e845.6303 0.3 12.1(19r)EW 12.2(18)EW, Ok

3 0005.9a80.3c00 to 0005.9a80.3c2f 0.9 Ok

System Failures:

----------------

Power Supply: bad/off (see 'show power')

RST-3508 9805_05_2004_c2

171717

Supervisor Redundancy

cat4507R#sh mod

Chassis Type : WS-C4507R

Power consumed by backplan e : 40 Watts

Mod Ports Card Type Model Serial No.

---+-----+---------------- ----------------------+------------------+-----------

1 2 1000BaseX (GBIC) Su pervisor(active) WS-X4013+ JAB071904FP

2 Standby Supervisor

3 48 10/100BaseTX (RJ45)V, Cisco/IEEE WS-X4248-RJ45V JAB074005BE

M MAC addresses Hw Fw Sw Status

--+----------------------- ---------+---+------------+----------------+---------

1 0009.e845.6300 to 0009.e845.6301 0.3 12.1(20r)EW 12.2(18)EW, EARL Ok

2 Unknown Unknown Unknown Other

3 0005.9a80.3c00 to 0005.9a80.3c2f 0.9 Ok

System Failures:

----------------

Power Supply: bad/off (see 'show power')

RST-3508 9805_05_2004_c2

181818

Power Supply Redundancy

cat4500(config)#power redundancy-mode ?

combined combine power supply outputs (no redundancy)

redundant either power supply can operate system (redundancy)

RST-3508 9805_05_2004_c2

Power Supply Redundancy

cat4507R#show power

Power Fan Inline

Supply Model No Type Status Sensor Status

------ ---------------- --------- ----------- ------- -------

PS1 PWR-C45-2800AC AC 2800W bad/off bad/off bad/off

PS2 PWR-C45-1400AC AC 1400W good good n.a.

*** Power Supplies of different type have been detected***

Power supplies needed by system : 2

Power supplies currently available : 1

*** Insufficient power supplies present for specified configuration ***

Power Summary Maximum

(in Watts) Used Available

---------------------- ---- ---------

System Power (12V) 335 1360

Inline Power (-50V) 0 0

Backplane Power (3.3V) 40 40

---------------------- ----

Total Used 375 (not to exceed Total Maximum Available = 1400)

RST-3508 9805_05_2004_c2

191919

202020

HSRP Redundancy

• HSRP configured between supervisors on

different Catalyst 4500 chassis

• HSRP configured between a Catalyst 4500

and an external router

• No HSRP between two supervisors on the

same chassis since the standby supervisor is in suspended mode

RST-3508 9805_05_2004_c2

Port Channel Redundancy

• Layer 2 EtherChannel

All interfaces in the same VLAN or trunks with trunking mode the same on both ends

When trunking…allowed ranges of VLANs must be the same

• Layer 3 EtherChannel bundles

Port channel must be created first; then...

Use “no switchport” to create the layer 3 ports in the channel

• Supervisor Engine ports in an EtherChannel

Supervisor II: Both ports can be in the EtherChannel

Single Cisco IOS supervisor in slot 1 or 2: gi1/1–2 or gi2/1–2 active

With Sup II-Plus or IV Dual Supervisors, only gig1/1 AND gig2/1 active; but gig1/2 and gig2/2 can be placed in an EtherChannel bundle for backup

With Supervisor V, all four uplinks are active

®

bundles

212121

RST-3508 9805_05_2004_c2

222222

Channel Troubleshooting

Commands Similar to the PAgP Commands Are Available for LACP

r3_4506#sh etherchannel summary(truncated output) Flags: D - down P - in port-channel

I - stand-alone s - suspended

R - Layer3 S - Layer2

U - in use

Number of channel-groups in use: 2 Number of aggregators: 2

Group Port-channel Protocol Ports

------+-------------+-----------+--------------------------1 Po1(SU) PAgP Gi1/1(P) Gi1/2(P) 2 Po2(RU) PAgP Fa3/46(P) Fa3/47(P)

r3_4507R_S4#sh int gig1/1 etherchannel (truncated) Port state = Up Cnt-bndl Suspend Not-in-Bndl Flags: S - Device is sending Slow hello. C - Device is in Consistent state.

A - Device is in Auto mode. P - Device learns on physical port. d - PAgP is down.

Local information:

Port Flags State Timers Interval Count Priority Method Ifindex Gi1/1 d U1/S1 1s 0 128 Any 0

Age of the port in the current state: 27d:19h:18m:59s

Probable reason: Source monitor interfaces are not allowed to be part of an etherchannel

Hello Partner PAgP Learning Group

Ages Every 30 Secs

RST-3508 9805_05_2004_c2

r3_4506 #sh int port-channel 1 ( truncated)

Port-channel1 is up, line protocol is up (connected)

Description: to cat4507R

MTU 1500 bytes, BW 2000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex, 1000Mb/s, media type is N/A

Members in this channel: Gi1/1 Gi1/2

r3_4506(config)#port-channel load-balance ?

r3_4506sh etherchannel load-balance Source XOR Destination IP address

r3_4506#sh pagp neighbor Flags: S - Device is sending Slow hello. C - Device is in Consistent state.

A - Device is in Auto mode. P - Device learns on physical port.

Channel group 1 neighbors

Partner Partner Partner Partner Group Port Name Device ID Port Age Flags Cap. Gi1/1 r3_4507R_S4.cisco.co 0009.e845.5f00 Gi1/1 17s SC 10001 Gi1/2 r3_4507R_S4.cisco.co 0009.e845.5f00 Gi2/1 24s SC 10001

Global Command

dst-ip Dst IP Addr dst-mac Dst Mac Addr dst-port Dst TCP/UDP Port src-dst-ip Src XOR Dst IP Addr src-dst-mac Src XOR Dst Mac Addr src-dst-port Src XOR Dst TCP/UDP Port src-ip Src IP Addr src-mac Src Mac Addr src-port Src TCP/UDP Port

232323

HARDWARE AND RELATED ISSUES

RST-3508 9805_05_2004_c2

242424

Switch Management Interfaces Cisco Catalyst OS

Cat4K-c (enable) sh int

sl0: flags=50<DOWN,POINTOPOINT,RUNNING>

slip 0.0.0.0 dest 0.0.0.0

sc0: flags=63<UP,BROADCAST,RUNNING>

vlan 1 inet 1.1.1.3 netmask 255.255.255.0 broadcast 1.1.1.255

me1: flags=62<DOWN,BROADCAST,RUNNING>

inet 0.0.0.0 netmask 0.0.0.0 broadcast 0.0.0.0

• sc0 inband management interface

• sc0 connects to switching fabric

• sc0 participates in STP, CDP, VLAN membership

• sl0 and me1 out-of-band management interfaces

• sl0 and me1 do not connect to switching fabric

• sl0 and me1 do not participate in STP, CDP, VLAN membership

• Only one out of sc0 and me1 can be up

RST-3508 9805_05_2004_c2

252525

Switch Ports/Interfaces

• On Cisco Catalyst OS switches these are

Layer 2 ports

• On Cisco IOS switches these can be

Layer 3 routed interfaces

Layer 3 Switched Virtual interfaces (SVIs)

Layer 3 portchannel interfaces

Layer 2 switchport interfaces—access or trunk

Layer 2 portchannel interfaces

• By default on Cisco IOS switches the interfaces are

Layer 2 switchport interfaces

• “no switchport” command converts these to Layer

3 routed interfaces

RST-3508 9805_05_2004_c2

262626

High CPU Usage-Supervisor II

Console> (enable) show proc cpu

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

--- ----------- ---------- -------- ------- ------- ------- --- ---------------

(truncated)

98 23438905 7904296 9352 86.64% 89.57% 87.50% 0 Switching overhead 99 2271479 1443242 57968 1.19% 1.04% 0.98% 0 Admin overhead

(truncated)

Remember: Kernel and Idle CPU Usage Is the Percentage of Time the CPU Was Idle

Console> (enable) sh proc cpu

(truncated)

CPU utilization for five seconds: 14.45%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

--- ----------- ---------- -------- ------- ------- ------- --- --------------1 345976604 0 0 85.55% 85.00% 85.00% -2 Kernel and Idle

RST-3508 9805_05_2004_c2

one minute: 15.00%

five minutes: 15.00%

272727

High CPU Usage-Supervisor II

• Switching overhead

Address learning (path setup) for new MAC addresses

Normal host entry aging, as well as fast aging due to reception of STP topology Change notification

Packet processing for control traffic such as STP BPDUs, CDP, VTP, DTP, PAgP, and so forth

Packet processing for management traffic such as telnet, SNMP, and HTTP

• Admin overhead

Switch fabric Application Specific Integrated Circuit (ASIC) and other hardware management

Line card ASIC management

Port monitoring

RST-3508 9805_05_2004_c2

282828

Cisco IOS Supervisor CPU Usage

Total CPU Utilization (Process + Interrupt)

cat4500# sh proc cpu CPU utilization for five seconds: 73%/17%; one minute: 74%; five minutes: 76% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

1 2358396 1705816 1382 0.32% 1.17% 0.68% 0 IP-EIGRP Router 2 2337276 21210 110196 0.00% 0.80% 0.89% 0 Check heaps 8 6551276 3786002 1730 3.05% 4.41% 4.70% 0 IP Input

9 24211844 1644250 14725 27.91% 26.06% 25.45% 0 22 15663744 474459 33014 19.71% 20.67% 21.89% 0 TCP Driver 32 508 36 14111 5.07% 0.73% 0.15% 13 Virtual Exec

Interrupt level

Average

Interrupt level

Processing Time

• Make sure to distinguish interrupt and process level

• A CPU utilization value of 20% to 50% is normal, even under

minimal load with Power over Ethernet (PoE) line cards

RST-3508 9805_05_2004_c2

TCP Timer

292929

CPU Troubleshooting Commands: Cisco IOS Supervisor

cat4500# sh proc cpu CPU utilization for five seconds: 99%/0%; one minute: 27%; five minutes: 15%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

(truncated)

25 1599072 5303348 301 10.01% 9.20% 8.20% 0 Cat4k Mgmt HiPri 26 1869444 522959 3574 86.16% 78.67% 36.07% 0 Cat4k Mgmt LoPri

• The Mgt Hi-Priority and Mgt Low-Priority are two processes

that Cisco Catalyst 4500 platform code runs

• These two processes share total CPU usage among other

Cisco IOS processes

The reason there are two processes is when a job in Cisco Catalyst 4500 platform takes longer time than expected or exceeds the max allocated CPU time, the job would be run under low-priority for sometime until it's usage is reduced, this would give other higher priority (IOS) process a chance to run

• If the above two processes consume most of the CPU time

further investigation is now needed

RST-3508 9805_05_2004_c2

303030

High CPU Usage: Cisco IOS Supervisors

• CPU usage is not indicative of forwarding

performance on any supervisor

• Forwarding decisions are made in hardware

• Packets sent to the CPU

Control packets: STP, CDP, PAgP, LACP, UDLD Routing protocol updates IPX/Appletalk SNMP/telnet/ ARP responses to ARP requests Packets with IP options/expired TTL or non-ARPA encaps Packets with special handling, i.e. tunneling, encryption ACL logging enabled Input ACL drops MTU check failure Adjacency same interface

RST-3508 9805_05_2004_c2

313131

Extras: QoS on the CPU Port

0: ESMP

1: Control

Packets to the CPU

…

CPU Queues

• Protects important traffic when CPU usage is high

BPDUs/routing updates get priority

RST-3508 9805_05_2004_c2

Can still telnet or SNMP query when CPU is high

2: Host Learning 3–5: L3 Forwarding

9–10: L3 Rx (Telnet/SNMP)

15: MTU Fail/Invalid

323232

Show Platform CPU Statistics Fields:

• ESMP:

Even Simpler Management Protocol…used by the CPU for reading line card status: link, speed, led, etc.

• Control:

L2 control plane packets go here—STP, CDP, PaGP, LACP, UDLD, etc.

• Host learning:

Packets with unknown L2 source address are copied to CPU to build CAM table

• L3 fwd:

GRE tunnels Gleaning

• L2 fwd: Any non-IP switchable packet

IPX/Appletalk Zero TTL field Non-ARPA encapsulated packets ARP’s

• L3 Rx:

L3 packets to the switch—SNMP, telnet, ping

• ACL forward

RST-3508 9805_05_2004_c2

EIGRP/OSPF updates

333333

CPU Troubleshooting Commands: Cisco IOS Supervisor

cat4500# show platform cpu packet statistics (all)>>> lots of output, look for Total packet queues 16

Packets Received by Packet Queue

Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg

---------------------- --------------- --------- --------- --------- ---------Esmp 42808 38 38 34 6 Control 9919 11 10 8 1 Host Learning 39 0 0 0 0 L3 Fwd High 0 0 0 0 0 L3 Fwd Medium 0 0 0 0 0 L3 Fwd Low 0 0 0 0 0 L2 Fwd High 0 0 0 0 0 L2 Fwd Medium 0 0 0 0 0 L2 Fwd Low 99929 0 5 92 17 L3 Rx High 0 0 0 0 0 L3 Rx Low 36 0 0 0 0 RPF Failure 0 0 0 0 0 ACL fwd(snooping) 1165 1 1 1 0 ACL log, unreach 0 0 0 0 0 ACL sw processing 0 0 0 0 0 MTU Fail/Invalid 0 0 0 0 0

L2 and L3 High, Medium, and Low Are Based on the DSCP/COS Field of the Packet

RST-3508 9805_05_2004_c2

343434

CPU Troubleshooting Commands: Cisco IOS Supervisor

Look for Received Packets and Rx Drops

cat4500# sh platform cpu packet driver ( truncated)

Queue rxTail received all guar allJ gurJ rxDrops rxDelays 0 Esmp 63A6B70 25708 100 100 0 5 0 0 1 Control 63A6CF4 5405 595 600 0 5 0 0 2 Host Learning 63A76A0 24 500 500 0 5 0 0 3 L3 Fwd High 63A7E10 0 300 300 0 5 0 0 4 L3 Fwd Medium 63A82C0 0 500 500 0 5 0 0 5 L3 Fwd Low 63A8A90 0 900 900 0 5 0 0 6 L2 Fwd High 63A98A0 0 300 300 0 5 0 0 7 L2 Fwd Medium 63A9D50 0 500 500 0 5 0 0

8 L2 Fwd Low 63AB2E4 99929 899 900 0 5 434063 0

RST-3508 9805_05_2004_c2

Monitor the CPU on Cisco Catalyst 4500

• Incoming packets are allowed on SPAN destination port

• Monitor the CPU port (excellent in capturing high CPU util)

• Cisco Catalyst 4500: Only LAN Switch where you can monitor on the CPU port

MAC

Table

FFE

CPU

Source Port/

VLAN

RST-3508 9805_05_2004_c2

Monitor

Destination

Port

Modules

Input

Packet

Enable

Inpkts

Switch Fabric

PPE

Packet

Memory

Mirror Source Ports, VLANs, CPU With

SPAN Capabilities on Catalyst 4500

ACL

TCAM

353535

363636

Monitoring the CPU on Cisco Catalyst 4500 IOS Supervisors

r3_4506 #sh monitor session 1 Session 1

--------Type : Local Session Source Ports :

RX Only : CPU(Queues: 32)

Destination Ports : Fa3/37

Encapsulation : Native

Ingress : Disabled

Learning : Disabled

RST-3508 9805_05_2004_c2

C(config)#monitor session 1 source cpu queue ?

<1-32> SPAN source CPU queue numbers

acl Input and output ACL [13-20]

adj-same-if Packets routed to the incoming interface

[7]

all All queues [1-32]

bridged L2/bridged packets [29-32]

control-packet Layer 2 Control Packets [5]

mtu-exceeded Output interface MTU exceeded [9]

nfl Packets sent to CPU by netflow (unused) [8]

routed L3/routed packets [21-28]

rpf-failure Multicast RPF Failures [6]

span SPAN to CPU (unused) [11]

unknown-sa Packets with missing source address [10]

373737

Ping Latency

• Low priority task on the CPU

• Response times of 7–10 ms are typical on

an idle switch

• Pings through the switch are handled as

ordinary data packets and switched in HW

RST-3508 9805_05_2004_c2

383838

Best Practices

• Baseline the CPU in steady state

Normally which processes are causing the highest CPU usage

• When troubleshooting

Are high CPU processes different from the baseline?

Is the CPU consistently elevated or just spiking?

Are there TCNs in the network caused by flapping ports?

Is there excessive broadcast or multicast traffic in the management subnet or VLAN?

Is there excessive management traffic such as SNMP polling?

• Isolate the management VLAN from VLANs with

user data traffic

Particularly heavy broadcast traffic such as IPX or AppleTalk

RST-3508 9805_05_2004_c2

Cisco Catalyst 4500 Architecture

• 3-slot, 6-slot chassis—one supervisor

with two or five module slots

• 7-slot chassis—one or two

supervisors with five module slots

• 10-slot chassis—one or two

supervisors (Supervisor V only) with eight module slots

• Cisco IOS supervisors provide:

Central forwarding engine (Fast Forwarding Engine, FFE)

Buffering and 64 Gbps fabric (Packet Processing Engine, PPE)…96 Gbps fabric with Supervisor V and PPE2

• 12 Gbps bandwidth to each module

• Auto MDIX on 10/100/1000 Ports

• Modules are transparent:

Contain simple “stub” ASICs, PHYs

RST-3508 9805_05_2004_c2

No buffering or local switching

Switching

Module

Note: Supervisor Engine V Supports 3 Additional Line Card Slots

Forwarding Engine (FFE)

Shared Memory

Fabric (PPE)

12 Gbps

Switching

Module

12 Gbps

Switching

Module

12 Gbps

Switching

Module

12 Gbps

393939

12 Gbps

Switching

Module

404040

Blocking and Non-Blocking GigE Ports

• A port that does not oversubscribe

access to the switching fabric is a nonblocking port

• A port that oversubscribes access to the

switching fabric is a blocking port

RST-3508 9805_05_2004_c2

Cisco Catalyst 4000/4500 Linecards

• Six full-duplex GbE connections to switch fabric

• Transparent

No local forwarding—all packets go to supervisor

• GbE connections from switch fabric straight to

front-panel port or connect to stubs

Supervisor Switch Fabric

Six Full-Duplex Gbps Connections to Supervisor Switch Fabric

414141

RST-3508 9805_05_2004_c2

424242

Stub ASIC Overview

• Fans out GigE ports from

switch fabric

• Up to 8 front-panel ports;

10/100, 1000-only, or 10/100/1000

• Flow control on gigabit

interfaces

• Ports can be used in an

EtherChannel

• Not always oversubscribed

e.g. 10/100

RST-3508 9805_05_2004_c2

GbE to/from

Switch Fabric

Up to 8 Front-Panel

Ports, 10/100/1000

434343

IEEE 802.3x Flow Control

• Standards-based mechanism used to control data flow

• Basic steps

1) Data flows to switch

2) Switch congested so “pause” frame sent

3) End station waits required time before sending

4) IOS supervisors support both Tx and Rx pause frames

Gigabit Ethernet

Switch

1. Data Flows to Switch

Port Rx-No-Pkt-Buff RxPauseFrames TxPauseFrames PauseFramesDrop Gi4/7 0 35648 4854 0

RST-3508 9805_05_2004_c2

2. Switch Congested “Pause” Frame Sent

3. End Station Waits Required Time Before Sending

File Server

444444

Blocking and Non-Blocking Ports

Non-Blocking Gigabit Line Cards

Supervisor Uplink Ports

WS-X4306-GB—All Ports

Two 1000 Base-X Ports on the

WS-X4232-GB-RJ

First Two Ports on WS-X4418-

GB

WS-X4302-GB—Both Ports

• Oversubscribed GbE modules are ideal

for deployments that are more bursty in nature such as Gigabit to the Desktop and Servers

• These interfaces are not recommended

for uplinks or sustained connections

RST-3508 9805_05_2004_c2

Blocking Gigabit

Line Card

All Ports on the WS-

X4424-GB-RJ45

X4448-GB-RJ45

All Ports on the WS-

X4448-GB-LX

Last 16 Ports on the WS-

X4418-GB

1000 BT Ports on the WS-

X4412-2GB-TX

All ports on the WS-

X4424-GB-RJ45

Oversubscription

Ratio for Blocking

Line Cards

4:1

8:1All Ports on the WS-

8:1WS-X4548-GB-RJ45V

8:1

4:1

454545

Dot 1Q/ISL/Jumbo Frame Support:

• Dot 1Q is supported on all ports: Non blocking

and stub ASIC

• With Supervisor II+/IV/V ISL is supported on all

linecards except

WS-X4418-GB (ISL on ports 1 and 2 only)

WS-X4412-2GB (ISL on ports 13 and 14 only)

• Supervisor I/II

ISL only on front panel gig E ports of WS-X4232-L3

• Jumbo frames are supported on non-blocking

ports and only on IOS supervisors

RST-3508 9805_05_2004_c2

464646

L2 Forwarding Tables to Verify Reachability

cat4500#show mac-address-table dynamic Unicast Entries

vlan mac address type protocols port

-------+---------------+--------+---------------------+-------------------1 0000.0c07.ac01 dynamic ip FastEthernet3/37 1 0009.e845.64bf dynamic ip FastEthernet3/37 1 0030.7b4e.340a dynamic ip,assigned FastEthernet3/37

41 0000.0c07.ac29 dynamic ip FastEthernet3/19 50 0000.0c07.ac32 dynamic ip FastEthernet3/19 50 000a.4172.df7f dynamic ip FastEthernet3/19

cat4500#show mac-address-table address 0000.0c07.ac29 Unicast Entries

vlan mac address type protocols port

-------+---------------+--------+---------------------+--------------------

41 0000.0c07.ac29 dynamic ip FastEthernet3/19

cat4500#show mac-address-table count MAC Entries for all vlans: Dynamic Unicast Address Count: 6 Static Unicast Address (User-defi ned) Count: 0 Static Unicast Address (System-de fined) Count: 1 Total Unicast MAC Addresses In Use: 7 Total Unicast MAC Addresses Available: 32768 Multicast MAC Address Count: 11 Total Multicast MAC Addresses Available: 16384

RST-3508 9805_05_2004_c2

474747

Show Interface Error Counters

• FCS-err is the number of valid size frames with FCS (frame check sequence) errors but no

framing errors; this is typically a physical issue (cabling, bad port, NIC card,…) but can also indicate a duplex mismatch

• Align-err is the number of frames with alignment errors (frames that do not end with an even

number of octets and have a bad CRC) received on the port; these usually indicate a physical problem (cabling, bad port, NIC card,…) but can also indicate a duplex mismatch; when the cable is first connected to the port, some of these errors may occur; also, if there is a hub connected to the port then collisions between other devices on the hub may cause these errors

• Late-coll (late collisions) is the number of times that a collision is detected on a particular port

late in the transmission process; for a 10Mbit/s port this is later than 512 bit-times into the transmission of a packet; five hundred and twelve bit-times corresponds to 51.2 microseconds on a 10 Mbit/s system; this error can indicate a duplex mismatch among other things; for the duplex mismatch scenario the late collision would be seen on the half duplex side; as the half duplex side is transmitting, the full duplex side does not wait its turn and transmits simultaneously causing a late collision; late collisions can also indicate an Ethernet cable/segment that is too long; collisions should not be seen on ports configured as full duplex

• Single-coll (single collision) is the number of times one collision occurred before the port

transmitted a frame to the media successfully; collisions are normal for port configured as half duplex but should not be seen on full duplex ports; if collisions are increasing dramatically this points to a highly utilized link or possibly a duplex mismatch with the attached device

• Multi-coll (multiple collision) is the number of times multiple collisions occurred before the port

transmitted a frame to the media successfully; collisions are normal for port configured as half duplex but should not be seen on full duplex ports; if collisions are increasing dramatically this points to a highly utilized link or possibly a duplex mismatch with the attached device

RST-3508 9805_05_2004_c2

484848

Show Interface Error Counters (Cont.)

• Excess-coll (excessive collisions) is a count of frames for which transmission on a

particular port fails due to excessive collisions; an excessive collision happens when a packet has a collision 16 times in a row; the packet is then dropped; excessive collisions is typically an indication that the load on the segment needs to be split across multiple segments but can also point to a duplex mismatch with the attached device; collisions should not be seen on ports configured as full duplex

• Carri-sen (carrier sense) occurs everytime an Ethernet controller wants to send data on a

half duplex connection; the controller senses the wire and check if it is not busy before transmitting; this is normal on an half-duplex Ethernet segment

• Undersize are frames received that are smaller than the minimum IEEE 802.3 frame size

of 64 bytes long (excluding framing bits, but including FCS octets) that were otherwise well formed; check the device sending out these frames

• Runts are frames received that are smaller than the minimum IEEE 802.3 frame size

(64bytes for Ethernet) and with a bad CRC; this can be caused by duplex mismatch and physical problems like a bad cable, port, or NIC card on the attached device

• Giants exceed the maximum IEEE 802.3 frame size (1518 bytes for non-jumbo Ethernet);

try to find the offending device and remove it from the network

• http://www.cisco.com/warp/public/473/164.html#show_interface

RST-3508 9805_05_2004_c2

Useful Port Troubleshooting Commands for Connectivity

r3_c4k_s2> (enable) sh port 3 (Cat OS) * = Configured MAC Address

Port Name Status Vlan Level Duplex Speed Type

----- ------------------ ---------- ---------- ------ ------ ----- -----------3/1 connected 201 normal full 100 10/100BaseTX 3/4 notconnect 20 normal auto auto 10/100BaseTX

Port States:

1. Connected: Operational

2. Not connected: Check cables

3. Faulty: Use a sh test mod #

4. Disabled: Admin down

5. Inactive: Typically VLAN doesn’t exist

6. Errdisable: EtherChannel mismatch; duplex mismatch causing

excessive late collisions; UDLD; BPDU Guard

494949

RST-3508 9805_05_2004_c2

505050

Useful Port Troubleshooting Commands for Performance and Connectivity

cat4003> (enable) sh mac 2/1 ( CAT OS)

Port Rcv-Unicast Rcv-Multicast Rcv-Broadcast

-------- -------------------- -------------------- --------------------

2/1 100999222 91857174 460433

Port Xmit-Unicast Xmit-Multicast Xmit-Broadcast

-------- -------------------- -------------------- --------------------

2/1 51713414 26520362 32

Port Rcv-Octet Xmit-Octet

-------- -------------------- --------------------

2/1 132521131606 96814952585

MAC Dely-Exced MTU-Exced In-Discard Lrn-Discrd In-Lost Out-Lost

-------- ---------- ---------- ---------- ---------- ---------- ----------

2/1 0 0 0 0 87 419821

Last-Time-Cleared

--------------------------

Thu Mar 20 2003, 12:09:25

Useful to Verify Traffic Flow Through an Interface

• In-discards: Traffic on a trunk VLAN but no switchports in the VLAN or if trunk is blocking

• In-lost: Packets dropped in the Receive Path; Rx-No-Packet Buffer Avail, Rx Crc Error, Rx Fragments, etc

• Out-lost: Output buffer is full...oversubscription of the output port

• Lrn-discard: Not able to learn a MAC address due to CAM table full or hash index collision

RST-3508 9805_05_2004_c2

515151

Useful Port Troubleshooting Commands for Performance

cat4003> (enable) show port counters 2/1 (CAT OS)

Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize

----- ---------- ---------- ---------- ---------- ---------

2/1 - 2 419824 2 0

Port Single-Col Multi-Col l Late-Coll Excess-Col Carri-Sen Runts Giants

----- ---------- ---------- ---------- ---------- ---- ----- --------- ---------

2/1 0 0 20215 0 1 0 0

Last-Time-Cleared

--------------------------

Thu Mar 20 2003, 12:09:25

Error Rate Should Be Less Than 3% of Traffic

• FCS_err: Due to bad CRC…faulty NIC or cable

• Xmit-err: Internal transmit buffer is full; oversubscription

• Rcv-err: Rx buffer is full

• Late collisions: Duplex mismatch

RST-3508 9805_05_2004_c2

525252

Port Troubleshooting Commands for Connectivity and Performance

cat4500#sh int gig 4/7 (IOS) GigabitEthernet4/7 is up, line protocol is up (connected)

Hardware is Gigabit Ethernet Port, address is 0009.e845.5f3f (bia 0009.e845.5f3f) Internet address is 10.17.1.1/24 MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 12/255, rxload 6/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex, 1000Mb/s, link type is auto, media type is SX output flow-control is off, input flow-control is off ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:46:11, output never, output hang never Last clearing of "show interface" counters 00:00:59 Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1234242 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 48766000 bits/sec, 82367 packets/sec L3 in Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes L3 out Switched: ucast: 20000000 pkt, 1120000000 bytes - mcast: 0 pkt, 0 bytes

0 packets input, 37973544 bytes, 0 no buffer

Received 0 broadcasts (0 IP multicast)

0 runts, 0 giants, 0 throttles

513156 input errors, 513156 CRC, 0 frame, 0 overrun, 0 ignored

0 input packets with dribble condition detected

18765774 packets output, 1388667646 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier

0 output buffer failures, 0 output buffers swapped out

If Down or Faulty Check Physical Connection Sh Diagnostic Result Module #

Oversubscription

Is Traffic Flowing

RST-3508 9805_05_2004_c2

Input Errors: CRC; Alignment Errors

Port Troubleshooting Commands for Performance

cat4500#sh int gig 4/7 counters detail

(truncated)

Port InBytes InUcastPkts InMcastPkts InBcastPkts Gi4/7 146 1 1 0

Port OutBytes OutUcastPkts OutMcastPkts OutBcastPkts Gi4/7 1078309438 14563961 5579 5

Port InPkts 64 OutPkts 64 InPkts 65-127 OutPkts 65-127 Gi4/7 1 1 1 14569125

Port InPkts 128-255 OutPkts 128-255 InPkts 256-511 OutPkts 256-511 Gi4/7 0 0 0 415

Port InPkts 512-1023 OutPkts 512-1023 Gi4/7 0 4

Port InPkts 1024-1522 OutPkts 1024-1522 InPkts 1523-1600 OutPkts 1523-1600 Gi4/7 0 0 0 0

Port Tx-Bytes-Queue-1 Tx-Bytes-Queue-2 Tx-Bytes-Queue-3 Tx-Bytes-Queue-4 Gi4/7 783454686 0 294281044 573772

Port Tx-Drops-Queue-1 Tx-Drops-Queue-2 Tx-Drops-Queue-3 Tx-Drops-Queue-4 Gi4/7 340119 0 43128 0

Port Dbl-Drops-Queue-1 Dbl-Drops-Queue-2 Dbl-Drops-Queue-3 Dbl-Drops-Queue-4 Gi4/7 0 0 0 0

Port Rx-No-Pkt-Buff RxPauseFrames TxPauseFrames PauseFramesDrop Gi4/7 0 35648 0 0

535353

Packet Size Distribution

Lots of Small

Packets: Could

Be DoS Attack

Which Q’s Are

Dropping

Flow Control Frames

RST-3508 9805_05_2004_c2

545454

Port Troubleshooting Commands for Performance

cat4500#sh int gig 4/3 counter errors

Port CrcAlign-Err Dropped-Bad-Pkts Collisions Symbol-Err Gi4/3 2225010 0 0 0

Port Undersize Oversize Fragments Jabbers Gi4/3 0 0 5947414 0

Port Single-Col Multi-Col Late-Col Excess-Col Gi4/3 0 0 0 0

Port Deferred-Col False-Car Carri-Sen Sequence-Err Gi4/3 0 0 0 0

• Symbol error: Physical problem with GBIC or fiber

• CRC/fragments: Physical layer or NIC

• Collisions: Typically duplex mismatch

RST-3508 9805_05_2004_c2

Documentation in a Show Command

cat4500#show interfaces capabilities module 3 FastEthernet3/1

Model: WS-X4248-RJ45V-RJ-45 Type: 10/100BaseTX Speed: 10,100,auto Duplex: half,full,auto Trunk encap. type: 802.1Q,ISL Trunk mode: on,off,desirable,nonegotiate Channel: yes Broadcast suppression: percentage(0-100), sw Flowcontrol: rx-(none),tx-(none) VLAN Membership: static, dynamic Fast Start: yes Queuing: rx-(N/A), tx-(1p3q1t, Shaping) CoS rewrite: yes ToS rewrite: yes Inline power: yes (Cisco Voice Protocol/IEEE Protoco l 802.3af) SPAN: source/destination UDLD: yes Link Debounce: no Link Debounce Time: no Port Security: yes Dot1x: yes Maximum MTU: 1552 bytes (Baby Giants)

(truncated)

555555

RST-3508 9805_05_2004_c2

565656

Trunk Troubleshooting

cat4500#sh int fa3/19 trunk

Port Mode Encapsulation Status Native vlan Fa3/19 on 802.1q trunking 1

Port Vlans allowed on trunk Fa3/19 1-4094

Port Vlans allowed and active in management domain Fa3/19 1-3,10,41,49-50,100-102,104

Port Vlans in spanning tree forwarding state and not pruned Fa3/19 1-3,10,41,49-50,100-102,104

cat4500# show interface fast 0/1 switchport Name: Fa0/1 Switchport: Enabled

Administrative Mode: trunk Operational Mode: trunk

Administrative Trunking Encapsulation: dot1q Operational Trunking Encapsulation: dot1q

Negotiation of Trunking: On

Access Mode VLAN: 1 (default) Trunking Native Mode VLAN: 1 (default) Voice VLAN: none Administrative private-vlan host-association: none Administrative private-vlan mapping: none Operational private-vlan: none

Trunking VLANs Enabled: ALL Pruning VLANs Enabled: 2-1001

Capture Mode Disabled

RST-3508 9805_05_2004_c2

Capture VLANs Allowed: ALL

cat4500#sh run int fa3/19 Building configuration...

Current configuration : 95 bytes ! interface FastEthernet3/19

switchport trunk encapsulation dot1q switchport mode trunk

end

Useful for STP Optimization Prune Unneeded VLANs

Check Operational State

575757

Spanning Tree Support

Cisco Catalyst OS and Cisco IOS Supervisor STP Support

• 802.1d Spanning Tree

• 802.1d PVST

• Uplink Fast

• Backbone Fast

• 802.1w(RST)/802.1s(MST)

• Rapid PVST+

RST-3508 9805_05_2004_c2

• Port fast

• Port fast BPDU Guard

• Port fast BPDU Filter

• Root Guard

• UDLD

585858

Spanning Tree Support

r3_4507R_S4# sh spanning-tree summary totals( sh spantree summary on CAT OS)

Switch is in pvst mode Root bridge for: VLAN0001-VLAN0003, VLAN0020 Extended system ID is enabled Portfast Default is disabled PortFast BPDU Guard Default is disabled Portfast BPDU Filter Default is disabled Loopguard Default is disabled EtherChannel misconfig guard is enabled UplinkFast is disabled BackboneFast is disabled Pathcost method used is short

Name Blocking Listening Learning Forwarding STP Active

---------------------- -------- --------- -------- ---------- ---------4 vlans 0 0 0 10 10

• Ensure the sum of the logical interfaces across all instances of Spanning Tree for

different VLANs does not exceed 3,000 for Supervisor IV/V, 1500 for Supervisor II-Plus;

Supervisor I and II support 400 PVST+ instances and 300 Rapid PVST+ instances

• Sum of logical interfaces = (# of trunks) x (# of active VLANs per trunk) + (non-trunking

interfaces)

• If greater use MST mode

RST-3508 9805_05_2004_c2

595959

UNICAST PACKET FORWARDING

RST-3508 9805_05_2004_c2

606060

Unicast Forwarding Example Topology

C4500 A

Fas 3/1 Fas 3/3

Gig 5/1 Gig 5/2

20.2.1.0 /24

10.6.1.2 Host A

RST-3508 9805_05_2004_c2

Unicast Forwarding: CEF

C4500 B

10.5.1.2 Host B

616161

SW

Routing ARP

FIB ADJ

HW

• Check the routing, ARP, CEF, ADJ info in the supervisor

• For both unicast and multicast, the SW and HW tables should always

be consistent...if not…hardware table is full

128k entries for Supervisor IV/V and 32k entries for Supervisor II-Plus

C4K_L3HWFORWARDING-2-FWDCAMFULL

HW adjacency table has 32K entries

C4K_L3HWFORWARDING-3-NOMOREK2FIBADJS

• If table is not full…possibly a hardware issue...contact TAC

RST-3508 9805_05_2004_c2

626262

Checking Hardware FIB Table

Apr 28 15:19:31.478 PDT: %C4K_L3HWFORWARDING-2-FWDCAMFULL: L3 routing table is full. Switching to software forwarding

4500#show platform hardware ip route summary ( truncated)

8169 blocks used out of 8192 (99.71%)

130245 K2Fib TCAM entries used out of 131072 (99.36%)

RST-3508 9805_05_2004_c2

Check the Routing Table (SW)

Cat4500 A # sh ip route 10.5.1.0

Routing entry for 10.5.1.0/24

Known via "eigrp 100", distance 90, metric 28672, type internal

Redistributing via eigrp 100

Last update from 20.2.1.2 on GigabitEthernet5/2, 00:23:23 ago

Routing Descriptor Blocks:

* 20.2.1.2, from 20.2.1.2, 00:23:23 ago, via GigabitEthernet5/2

Route metric is 28672, traffic share count is 1

Total delay is 120 microseconds, minimum bandwidth is 100000 Kbit

Reliability 255/255, minimum MTU 1500 bytes

Loading 1/255, Hops 2

Fas 3/1

C4500 A

Gig 5/1

Gig 5/2

20.2.1.0/24

C4500 B

636363

Fas 3/3

10.6.1.2

Host A

RST-3508 9805_05_2004_c2

10.5.1.2 Host B

646464

Check the FIB Table (HW)

Cat4500 A # sh ip cef 10.5.1.2 detail

10.5.1.0/24, version 963, epoch 0, cached adjacency

20.2.1.2

0 packets, 0 bytes

via 20.2.1.2, GigabitEthernet5/2, 0 dependencies

next hop 20.2.1.2, GigabitEthernet5/2

valid cached adjacency

RST-3508 9805_05_2004_c2

Host A

C4500 A

Fas 3/1

10.6.1.2

Gig 5/1

Gig 5/2

20.2.1.0/24

C4500 B

Fas 3/3

10.5.1.2 Host B

Check the ARP Table for Next Hop Neighbor (SW)

Cat4500 A # sh ip arp 20.2.1.2

Protocol Address Age (min) Hardware Addr Type Interface

Internet 20.2.1.2 233 000b.fdb3.9400 ARPA GigabitEthernet5/2

656565

C4500 A

Fas 3/1

10.6.1.2

Host A

RST-3508 9805_05_2004_c2

Gig 5/1

Gig 5/2

20.2.1.0/24

C4500 B

Fas 3/3

10.5.1.2 Host B

666666

Check the Adjacency Table (HW)

Cat4500 A# sh adjacency detail

Protocol Interface Address

IP GigabitEthernet5/2 20.2.1.2(19)

5099680 packets, 234585280 bytes

000BFDB39400000A4172E8BF0800

ARP 00:24:51

Epoch: 0

C4500 A

Fas 3/1

10.6.1.2

Host A

RST-3508 9805_05_2004_c2

MULTICAST PACKET FORWARDING

Gig 5/1

Gig 5/2

20.2.1.0/24

C4500 B

Fas 3/3

10.5.1.2 Host B

676767

RST-3508 9805_05_2004_c2

686868

World of Multicast

Multicast Routing

PIM

IGMP

Snooping

CGMP

IGMP

IGMP—Router ↔ Source/Receiver

CGMP—Router → Switch

IGMP Snooping—Switch Eavesdrops on IGMP

PIM—Router ↔ Router

RST-3508 9805_05_2004_c2

CGMP Outputs on Supervisor II: Cisco Catalyst OS

Console> (enable) show multicast protocols status

CGMP enabled

CGMP leave disabled

GMRP disabled

696969

Console> (enable) show cgmp leave

CGMP: enabled

CGMP leave: disabled

CGMP FastLeave: enabled

No IGMP Snooping Support on Sup I/II

RST-3508 9805_05_2004_c2

707070

Check Mcast Group and Mcast Router

Console> (enable) sh multicast group

VLAN Dest MAC/Route Des [CoS] Destination Ports or VCs /

[Protocol Type]

---- ------------------ ----- ----------------------------------

---------

4 01-00-5e-00-01-28 2/1

4 01-00-5e-01-01-01 2/1-2

Total Number of Entries = 2

Console> (enable) sh multicast router

Port Vlan

RST-3508 9805_05_2004_c2

-------------- ----------------

2/1 4

Total Number of Entries = 1

'*' - Configured

'+' - RGMP-capable

'#' - Channeled Port

A CGMP Server Is Required

717171

Cisco Catalyst 4500 IOS Supervisor Multicast Features

• For Cisco IOS Supervisor Engines

PIM-SM, PIM-DM, MSDP, MBGP, IGMP (v3), DVMRP, SSM (Source Specific Mcast)

(OSPF, EIGRP, BGP,...)

MBGP; MSDP; PGM supported on Supervisor IV and V only

• Full bridging feature set

IGMP snooping v1, 2, 3, CGMP server

(STP, SPAN, PAgP, private VLANs,...)

• Full QoS support for multicast, 4 queues per port

• All features done in hardware

RST-3508 9805_05_2004_c2

727272

Multicast Forwarding Example Topology

Catalyst 4507R A

Catalyst 4507R B

Switchport Fas 4/3 VLAN 3

10.1.3.100

VLAN 3

Routed Port Gig 4/1

202.202.202.100

Multicast Group

224.1.1.1

RST-3508 9805_05_2004_c2

Switchport Gig 1/1 VLAN 201

Multicast Source

201.201.201.1

Check IGMP Group to Verify the Receiver Has Joined the Multicast Group

737373

cat4507R B #show ip igmp group IGMP Connected Group Membership Group Address Interface Uptime Expires Last

224.0.1.40 Vlan3 03:16:16 00:02:50 10.1.3.1

224.1.1.1 Vlan3 00:00:03 00:02:56 10.1.3.100

224.1.1.1 GigabitEthernet4/1 00:00:39 00:02:20 202.202.202.100

Note: IGMP Report from a Receiver on a Port on VLAN 3 (Fas 4/3) IGMP Report from a Receiver on a Routed Port Gig E 4/1

Switchport Gig 1/1 VLAN 201

RST-3508 9805_05_2004_c2

Multicast Source

201.201.201.1

Reporter

Catalyst 4507R BCatalyst 4507R A

Switchport Fas 4/3

10.1.3.100 VLAN 3

Routed Port Gig 4/1

202.202.202.100 Multicast Group

224.1.1.1

747474

Check Multicast MAC Address

cat4507R B # show mac-address-table int fast4/3 Unicast Entries vlan mac address type protocols port

-------+---------------+--------+---------------------+-------------------3 0000.0000.1501 dynamic ip FastEthernet4/3

Multicast Entries vlan mac address type ports

-------+---------------+-------+------------------------------------------3 0100.5e01.0101 igmp Switch,Fa4/3

3 ffff.ffff.ffff system Switch,Fa4/3

The Entry We Are Looking for Has Fast 4/3 in the Port List

Catalyst 4507R BCatalyst 4507R A

Switchport Fas 4/3

10.1.3.100 VLAN 3

Routed Port Gig 4/1

202.202.202.100 Multicast Group

224.1.1.1

RST-3508 9805_05_2004_c2

Switchport Gig 1/1 VLAN 201

Multicast Source

201.201.201.1

Check Cisco IOS Multicast Routing Table (SW)

cat4507R B # show ip mroute 224.1.1.1 IP Multicast Routing Table Flags: D - Dense, S - Sparse, s - SSM Group, C - Connected, L - Local,

P - Pruned, R - RP-bit set, F - Register flag, T - SPT-bit set, J - Join SPT, M - MSDP created entry, X - Proxy Join Timer Running A - Advertised via MSDP, U - URD, I - Received Source Specific Host

Outgoing interface flags: H - Hardware switched Timers: Uptime/Expires Interface state: Interface, Next- Hop or VCD, State/Mode (*, 224.1.1.1), 00:14:24/00:02:59, RP 10.1.3.3, flags: SJC

Incoming interface: Vlan3, RPF nbr 10. 1.3.3 Outgoing interface list:

(201.201.201.1, 224.1.1.1) , 00:14:23/00:02:57, flags: CJT

Incoming interface: Vlan3, RPF nbr 1.1 .3.3

Outgoing interface list:

RST-3508 9805_05_2004_c2

Report

GigabitEthernet4/1, Forward/Sparse, 00:14:24/00:02:33, H

GigabitEthernet4/1,Forward/Sparse,00:14:23/00:02:33, H

Catalyst 4507R BCatalyst 4507R A

Switchport Gig 1/1 VLAN 201

Multicast Source

201.201.201.1

Switchport Fas 4/3

10.1.3.100 VLAN 3

Routed Port Gig 4/1

202.202.202.100 Multicast Group

224.1.1.1

757575

Note: FastEthernet 4/3 on VLAN 3 Is Not Listed as It Is an L2 Switchport

767676

Check MFIB Table (HW)

cat4507R B # show ip mfib 224.1.1.1 IP Multicast Forwarding Information Base Entry Flags: C - Directly Connected, S - Signal, IC - Internal Copy Interface Flags: A - Accept, F - Forward, S - Signal IC - Internal Copy, NP - Not Platform fast-switched Packets: Fast/Partial/Slow Bytes: Fast/Partial/Slow (*, 224.1.1.1), flags () Packets: 2708/1/0, Bytes: 124568/46/0 Vlan3 (A S) GigabitEthernet4/1 (F S) (201.201.201.1, 224.1.1.1), flags () <--check to see if the S,G entry exists

Packets: 20111339/1504/7, Bytes: 925121594/69184/322

Vlan3 (A) <---rpf vlan is correct

GigabitEthernet4/1 (F S)

RST-3508 9805_05_2004_c2

Only the first part should increment if it is fully HW switched

Gigabit 4/1 is correct and flag 'F' means forwarding is in HW FastEthernet 4/3 is not listed as it is a switchport and it is part of vlan 3 which is the incoming vlan. If the switchport was for eg on vlan4, then you would see vlan 4 in the OIF as a forwarding interface

Catalyst 4507R BCatalyst 4507R A

VLAN 3

Switchport Gig 1/1 VLAN 201

Multicast Source

201.201.201.1

Switchport Fas 4/3

10.1.3.100 VLAN 3

Routed Port Gig 4/1

202.202.202.100 Multicast Group

224.1.1.1

777777

ACLS

RST-3508 9805_05_2004_c2

787878

Different Types of ACLs

Mac Access Control List (MACL)

VLAN Access Map (VACL)

Port Access Control List (PACL)

Router Access Control List (RACL)

QOS Access Control Lists Control Traffic via Classification

and Policing Using Modular QOS CLI (MQC) Configuration

RST-3508 9805_05_2004_c2

Types of ACLs

Where Applied

L2 Switch Port

VLAN List

L2 Switch Port

L3 Switch Port or

SVI

L2 and Non-IP

and L3/4 IP

L3/L4 IP

DirectionTraffic ControlACL Type

Inbound

Outbound

Directionless

Inbound

Outbound

Inbound

Outbound

797979

RACL—

VLAN 10

VLAN 10 VLAN 20

VACL—

VLAN 10

PACL/MACL—

Fa 4/1

The above diagram is a logical model

RST-3508 9805_05_2004_c2

Switch

Router

RACL-

VLAN 20

VACL—

VLAN 20

Routed PacketBridged Packet

808080

ACL Hardware Programming

• TCAM: Ternary Content Addressable Memory

Value, Mask and Result values are used

Value and Mask used to identify L2/L3/L4 flows of interest

Result can be…permit or deny for security ACL

Result can be…classification or policing for QoS ACL

• Security and QoS ACLs get programmed into

dedicated TCAMs

• TCAM is a finite HW resource

• Advantage: ACLs are implemented in HW, therefore

no performance penalty

RST-3508 9805_05_2004_c2

Cisco IOS Supervisor ACL TCAM Details

Security

32000 Patterns

Supervisor

Engine

IV/V*

4000 Masks

QoS

32000 Patterns

4000 Masks

• Security ACLs TCAM is used for RACLs, VACLs, PACLs,

MAC-based ACLs, time of the day ACLs and security features like DHCP Snooping; Dynamic ARP Inspection and IP Source Guard

2 x 1 Banks of TCAM 1 x Used for QoS 1 x Used for Security ACLs

818181

• QoS TCAM is used for QoS functions: Classification,

Service Policies

*Supervisor Engine II-Plus as of IOS 12.2.18EW has 1/8 the TCAM entries

RST-3508 9805_05_2004_c2

828282

Applying a RACL/PACL

interface Vlan4

ip address 4.4.4.1 255.255.255.0

end

Counters

Done in

HW

cat4507R#show ip access-lists Extended IP access list 101 deny tcp host 200.200.200.1 any neq 80 (5 matches) permit ip any any (11915 matches)

Cat4507(config)#interface vlan 4

Cat4507(config-if)#ip access-group 101 in

Cat4507(config-if)#

Cat4507(config)#interface fa 4/23

Cat4507(config-if)#swichport access vlan 4

Cat4507(config-if)#ip access-group 101 in

RST-3508 9805_05_2004_c2

Layer 4 Operators (L4 Ops)

• The (operator, operand) tuples for TCP and UDP

port numbers

RACL

PACL

838383

• These ACL operators are considered L4 Ops:

gt

lt

neq

range

access-list 106 permit tcp any range 100 120 any range

120 140

• Greater than 6 L4 ops limit in an ACL…results in

ACE expansion or more TCAM entries being used

http://www.cisco.com/univercd/cc/td/doc/product/lan/cat4000/12_2_18/config/secure.htm#1050515

RST-3508 9805_05_2004_c2

848484

MAC ACLs

• MAC ACLs can be used to filter non-IP traffic

• MAC ACLs do not filter IP traffic

cat4507R# sh access-lists

Extended IP access list 101

permit ip host 4.4.4.3 any

Extended MAC access list decnet_acl

deny any any protocol-family decnet

permit any any

RST-3508 9805_05_2004_c2

VLAN ACL Map (VACL)

mac access-list extended drop-appletalk

permit any any protocol-family

appletalk

ip access-list extended ip2

permit ip any any

vlan access-map vacl-100 15

action drop

match mac address drop-appletalk

vlan access-map vacl-100 20

action forward

match ip address ip2

!

vlan filter vacl-100 vlan-list 201

858585

• VACLs match all

packets on the VLAN

• VACLs may have IP

based and MAC based ACLs, with implicit deny all at the end

• This example will

permit IP and drop all Appletalk frames on VLAN 201

RST-3508 9805_05_2004_c2

868686

ACL Main Issues

• High CPU

• Misbehaving ACLs

RST-3508 9805_05_2004_c2

ACL…High CPU

• Denied traffic in an input/output RACL

This is rate controlled starting in IOS 12.1.13 EW(1)

No effect on counter accuracy

Do not need “no ip unreachables” option with above release or

higher

• ACEs requiring logging (“log” keyword)

This is rate controlled starting in IOS 12.1.13 EW(1)

No effect on counter accuracy

• Match on TCP flags other than “established”

• Policy-routed traffic (SW switched for “set ip df”...ingress

packet size is greater than egress port MTU)

878787

• TCAM full due to excessive L4 ops expansion

RST-3508 9805_05_2004_c2

888888

Checking TCAM Usage

Apr 22 09:25:13.626 PDT: %C4K_HWACLMAN-4-ACLHWPROGERR: Input Security: 199 - hardware TCAM limit, some

packet processing will be software switched.

Apr 22 09:25:13.626 PDT: %C4K_HWACLMAN-4-ACLHWPROGERRREASON: Input Security: 199 - out of hardware TCAM entries.

r3_4507R_S4#sh platform hardware acl statistics utilization (truncated output from Supervisor II-Plus)

Software Usage Statistics

Input FeatureCam

PortAndVlan Entries 0 ( 0.0) 1024 (100.0) 1024 PortAndVlan Masks 0 ( 0.0) 128 (100.0) 128 PortOrVlan Entries 231 ( 22.5) 793 ( 77.4) 1024

PortOrVlan Masks 128 (100.0) 0 ( 0.0) 128

Output FeatureCam

PortAndVlan Entries 0 ( 0.0) 1024 (100.0) 1024

PortAndVlan Masks 0 ( 0.0) 128 (100.0) 1 PortOrVlan Entries 11 ( 1.0) 1013 ( 98.9) 1024 PortOrVlan Masks 11 ( 8.5) 117 ( 91.4) 128

Supervisor IV and V Have Larger TCAMs

• Input feature TCAM is used for security based features: PACL; RACL; DHCP

Snooping; Dynamic ARP Inspection; IP Source Guard

• Output feature TCAM is used for outbound RACLs and PACLs; DHCP

Snooping

RST-3508 9805_05_2004_c2

Used (%) Free (%) Total

-------------- -------------- ------

898989

Security ACL Feature TCAM

• Be careful when using L4 ops collapse contiguous

ranges into a single ACE if possible or use eq operator

• Check TCAM usage as ACLs being added

• Consider a Supervisor IV or V with larger TCAM

space than Supervisor II-Plus

• Mask allocation optimization is in latest IOS

release, 12.2.20 EW

RST-3508 9805_05_2004_c2

909090

When ACLs Are Misbehaving

ACLs Passing or Dropping Traffic when They Are Not Supposed to

• Remove ACL and see if drops are still there

• Check access-list counters

Use clear access-list counters command, and then check the statistics

with show access-list

Counters update every 15 seconds

If the packets are hitting some deny entry, then the packet will be

dropped…check your configuration

• Check interface counters to make sure that the box is indeed

receiving packets

• Remember implicit IP deny any any at the end of an ACL—

make it explicit

• Check CPU utilization

If packets are being processed in software...there can be drops

RST-3508 9805_05_2004_c2

919191

Miscellaneous ACL Considerations

• Fragments are being permitted

Layer 4 information is available only in the first fragment

• Fragments are being dropped

Tiny fragments are dropped to prevent DOS attacks

• TOS/DSCP fields are not being matched correctly

Check the trust state of the port

RST-3508 9805_05_2004_c2

929292

QoS

RST-3508 9805_05_2004_c2

QoS Terminology

• QoS labels are used to prioritize traffic

COS, TOS, DSCP

• Classification is selection of traffic based on

labels, policy

• Marking is application of QoS labels to traffic

• Policing is process by which the switch limits the

bandwidth consumed by a flow of traffic

• Queuing is placing of traffic in different

transmit queues

939393

• Scheduling is process of emptying the

transmit queues

RST-3508 9805_05_2004_c2

949494

Catalyst 4500 QoS Capabilities

Supervisor II (CAT OS)

• Layer 2 only

• System-wide QoS

• Dual queues per port

Queue 1

Queue 2

RST-3508 9805_05_2004_c2

Cisco IOS Supervisors

• Layer 2, 3, or 4 QoS

• Per-port QoS

• Four Tx queues per port

• Strict priority queue

• Dynamic queue memory

allocation

• Packet classification and

marking

• Policing/bursting

• Shaping/sharing

Queue 4

Queue 1

Queue 3

Queue 2

959595

QoS on Supervisor I/II (Catalyst OS)

• System-wide QoS configuration

• Global configuration applies to all ports on

the switch

• Disabling QoS configuration disables QoS on

all ports

• By default a port state is untrusted

• Ports can be set to have a default CoS on a

system-wide basis

RST-3508 9805_05_2004_c2

969696

Supervisor II QoS

Cat4K-c (enable) sh qos info runtime Run time setting of QoS:

QoS is disabled

Cat4K-c (enable) set qos enable

QoS is enabled.

Console> (enable) sh qos info runtime Run time setting of QoS:

QoS is enabled

All ports have 2 transmit queues with 1 drop thresholds (2q1t).

Default CoS = 0

Queue and Threshold Mapping: Queue Threshold CoS

----- --------- --------------1 1 0 1 2 3 4 5 6 7 2 1

Throughput Has Just

Been Halved!!

Must Re-Map CoS

Values to Queue 2

RST-3508 9805_05_2004_c2

Supervisor II QoS

Console> (enable) set qos map 2q1t 2 1 cos 4-7 Qos tx priority queue and threshold mapped to cos successfully.

Console> (enable) sh qos info runtime

Queue Threshold CoS

----- --------- --------------1 1 0 1 2 3 2 1 4 5 6 7

Cat4K (enable) set qos defaultcos 7 qos defaultcos set to 7 Cat4K (enable) Cat4k (enable) sh qos info runtime Run time setting of QoS: QoS is enabled All ports have 2 transmit queues with 1 drop thresholds (2q1t).

Default CoS = 7

Queue and Threshold Mapping: Queue Threshold CoS

----- --------- --------------1 1 0 1 2 3 2 1 4 5 6 7

RST-3508 9805_05_2004_c2

Re-Mapping CoS Values

to Queue 2

System Wide CoS

979797

Mapping

989898

Cisco IOS-Based Supervisor QoS Flow Summary

Policing via ACLs

Police Action:

Mark

Classification Based on:

Default DSCP Port Setting

Port “Trusted” CoS or DSCP

Layer 2/3/4 ACLs

Shared

RX

Memory

Classify

Drop

Based:

Byte Rate

Burst

(Token Bucket)

Police

DBL

Rewrite

Info

Sharing and Shaping

and Strict Priority Q3

to Schedule Between

Output Queues

Queue 1

Queue 2

Queue 3

Queue 4

Sched TX

In-Coming

Encapsulation

Can Be 802.1Q,

802.1p, ISL, or None

RST-3508 9805_05_2004_c2

Dynamic Buffer Limiting

(Supervisor II Plus

Supervisor IV

Supervisor V )

Congestion Avoidance

Rewrites TOS

Field in IP

Header and

802.1p/ISL CoS Field

Tx Queuing and Scheduling

Q1

Q2

DSCP to

Queue Map

Q3

Shaping

Q4

• Queue selection based on “internal DSCP”

Default DSCP on port Trust CoS/DSCP Via service policies

• Switch-wide DSCP to Tx queue map, not per-port!

• Shaping: Max rate per queue

• Sharing: Min rate per queue

• Strict priority on queue 3

• All in hardware at wire rate

RST-3508 9805_05_2004_c2

Strict Priority

Sharing,

Out-Going

Encapsulation

Can Be 802.1Q,

802.1p, ISL, or None

999999

Ethernet

MAC

100100100

Scheduling: Shaping

• Max rate (10K to 1 Gbps)

Shaped queue like a “virtual wire” Packets clock out exactly at shaped rate

• Hold packets in queue when rate exceeded

• Example use:

Shape a bursty application to 1 Mbps to smooth it

• Supported on all ports, typically used with strict

priority queue

Shaper (Specifies Max BW)

TX Port Q

RST-3508 9805_05_2004_c2

Scheduling: Sharing

• Minimum rate (32 Kbps to 1 Gbps)

Rate is guaranteed minimum

• Scheduling algorithm:

If below share rate, queue is high priority High priority queues serviced first

• Sharing only on non-blocking gigabit ports in Supervisor IV

and II-Plus

• Supported on ALL ports on Supervisor Engine V

Shaper (Specifies Max BW)

Non-Blocking Port TX Port Q

Sharer (Specifies Min Guaranteed BW)

101101101

RST-3508 9805_05_2004_c2

102102102

QoS Issues: First Check if QoS Is Enabled

By Default QoS Is Disabled and All Port Trust States Are Trusted

cat4500#show qos

QoS is enabled globally

IP header DSCP rewrite is enabled

RST-3508 9805_05_2004_c2

Check the Port

What Is the Port’s Trust State?

cat4500#show qos interface gig6/4

QoS is enabled globally

Port QoS is enabled

Port Trust State: 'dscp'

Default DSCP: 0 Default CoS: 0

Appliance trust: none

Tx-Queue Bandwidth ShapeRate Priority QueueSize

(bps) (bps) (packets)

1 250000000 disabled N/A 1920

2 250000000 disabled N/A 1920

3 250000000 50000000 high 1920

4 250000000 disabled N/A 1920

103103103

RST-3508 9805_05_2004_c2

104104104

Packet Classification and Marking

Check the Service-Policy

Qos

access-list 100 permit udp any any

!

class-map match-all class_setprec

match ip access group 100

!

policy-map pol_setprec

class class_setprec

set ip precedence 3

interface Vlan4

ip address 4.4.4.1 255.255.255.0

service-policy input pol_setprec

RST-3508 9805_05_2004_c2

Packet Classification and Marking

Is the Class Map Defined Properly?

Cat4500# show policy-map interface vlan 4

Interface vlan 4

service-policy input: pol_setprec

class-map: class_setprec (match-all)

0 packets match: ip access group 100 police: Per-interface

Conform: 0 bytes Exceed: 0 bytes

class-map: class-default (match-any)

32423 packets match: any

0 packets

105105105

No Packets Match! Check ACL

• Check class map statistics

• The packet statistics are on a per-class-map basis, and NOT

on a per-interface basis

RST-3508 9805_05_2004_c2

106106106

Check QoS TCAM

r3_4507R#sh platform hardware acl statistics utilization

( truncated output from Supervisor II-Plus)

Used (%) Free (%) Total

Input QosCam

PortAndVlan Entries 0 ( 0.0) 1024 (100.0) 1024 PortAndVlan Masks 0 ( 0.0) 128 (100.0) 128 PortOrVlan Entries 493 ( 48.1) 531 ( 51.8) 1024 PortOrVlan Masks 121 ( 94.5) 7 ( 5.4) 128

Output QosCAM

PortAndVlan Entries 0 ( 0.0) 1024 (100.0) 1024

PortAndVlan Masks 0 ( 0.0) 128 (100.0) 128 PortOrVlan Entries 0 ( 0.0) 1024 (100.0) 1024

PortOrVlan Masks 0 ( 0.0) 128 (100.0) 128

Note: Supervisor IV and V have larger TCAM’s

D-4500(config)#int fastEthernet 2/1 D-4500(config-if)# service-policy input classVOIP

D-4500(config-if)# 00:43:58: %C4K_HWACLMAN-4-ACLHWPROGERR: Input Policy Map: classVOIP - hardware TCAM limit, qos being disabled on relevant interface. 00:43:58:

%C4K_HWACLMAN-4-ACLHWPROGERRREASON: Input Policy Map: classVOIP -

RST-3508 9805_05_2004_c2

out of hardware TCAM entries.

------------- ---------- ---- ------

107107107

Optimize QoS TCAM Utilization

r3_4507R#sh platform hardware acl statistics utilization

With the default port based QoS

------------- -------------- -----PortOrVlan Entries 521 ( 50.8) 503( 49.1) 1024 PortOrVlan Masks 124 ( 96.8) 4( 3.1)128

With VLAN based QoS ..the same Service Policy is configured on a SVI...

PortOrVlan Entries 21 ( 2.0) 1003 ( 97.9)1024 PortOrVlan Masks 19 ( 14.8) 109 ( 85.1)128

• Use VLAN based QoS…if classification rules are the same within the VLAN

• General Policy should use VLAN based QoS…exceptions use port based QoS

QoS Policy on the

RST-3508 9805_05_2004_c2

Port

Service Policy

Applied to the VLAN

Used (%) Free (%) Total

Service Policy

Applied to the Port

Service Policy Used

VLAN BasedYesYesVLAN-Based

Port BasedYesYesPort-Based (Default)

108108108

COS/DSCP Mapping

Check if COS to DSCP and DSCP to COS Mapping Is Configured Properly

Cat4500(config)# qos map cos 7 to dscp 40

r3_4507R_S4#sh qos maps cos dscp CoS-DSCP Mapping Table

CoS: 0 1 2 3 4 5 6 7

-------------------------------DSCP: 0 8 16 24 32 40 48 40

• If a L2 trunk port is configured to "trust dscp",

If the packet is an IP packet, then it will use the DSCP from the IP header If not, it will use the port default DSCP (configured via qos dscp <val>)

• If a L2 trunk port is configured to "trust cos",

If the packet is tagged it will map the CoS to an internal DSCP (as per CoS-to-DSCP mapping table) and then use the DSCP-to-TxQ mapping to determine queue and the DSCP-to-CoS table to determine egress CoS

• If the packet is untagged, it uses the port default CoS and then the

other mapping tables as explained above

http://www.cisco.com/en/US/products/hw/switches/ps4324/products_configuration_guide_chapter09186a00801cddc9.html#1223900

RST-3508 9805_05_2004_c2

Cat4500(config)#qos map dscp 40 to cos 7 Cat4500#sh qos maps dscp cos( truncated) DSCP-CoS Mapping Table (dscp = d1d2) d1 : d2 0 1 2 3 4 5 6 7 8 9

------------------------------------4 : 07 05 05 05 05 05 05 05 06 06 5 : 06 06 06 06 06 06 07 07 07 07

109109109

Classification/TOS Re-Write Summary

• Determine the “internal DSCP” value—this

depends on the policy-map and port trust configuration

• If a packet encounters both input and

output classification policy:

Output policy has precedence

If no output policy then input policy has precedence

If no output/input policy then RX port trust is used

RST-3508 9805_05_2004_c2

110110110

Policing on the Supervisor II+/IV/V

• Two types of policers

Individual: Acts on each of the applied ports/VLAN

Aggregate: Acts on all of the applied ports/VLAN

• Two policer parameters: Rate and burst

Rate from 32kbps to 32gbps, burst in bytes

• Two actions

exceed-action: drop, transmit, markdown

conform-action: drop, transmit

• Input and output policing on every packet

1020 input, 1020 output policers, sharable: Sup IV/V

510 input, 510 output policers, sharable: Sup II Plus

RST-3508 9805_05_2004_c2

Policing Issues

Check QoS Policer Utilization

cat4500# show platform hardware qos policers utilization ( truncated)

Software Usage Statistics

Used (%) Free (%) Total

-------------- -------------- ------

Input Policers 4 ( 0.3) 1020 ( 99.6) 1024

Output Policers 5 ( 0.4) 1019 ( 99.5) 1024

111111111

*Above output from Supervisor IV/V; Supervisor II-Plus has half the entries

RST-3508 9805_05_2004_c2

112112112

Policing Issues

Make Sure the Correct Type of Policer Is Used

Cat4500# show policy-map interface Gig1/1

GigabitEthernet1/1

service-policy input: p1

class-map: c1 (match-all)

3435 packets match: access-group 100 police: Per-interface <----- This is a individual policer.

Conform: 45454 bytes Exceed: 56345 bytes

Cat4500# show policy-map interface Gig1/2

GigabitEthernet1/1

service-policy input: p1

class-map: c1 (match-all)

335 packets match: access-group 100 police: policer1 <----- This is an aggregate or named policer.

Conform: 4554 bytes Exceed: 563 bytes

RST-3508 9805_05_2004_c2

Policing Issues

Check the Service Policy

Qos

access-list 100 permit udp any any

!

class-map match-all class_udp

match ip access group 100

!

policy-map pol_udp

class class_udp

police 500 kbps 1000 byte conform-action transmit exceed-action

interface Vlan4

ip address 4.4.4.1 255.255.255.0

service-policy input pol_udp

Use the Show Policy-Map Interface Command to Check for Class Map Hits

RST-3508 9805_05_2004_c2

policed-dscp-transmit

113113113

114114114

Policing: How to Set the Burst Size?

• Too small: And policer drops due to burstiness inherent in all

networks

• Too large: And the entire transfer fits in the burst (especially for TCP)

• Small burst size [n*max pkt size] ok for video, voice

• Larger burst needed for TCP: 2 x [RTT * rate], good starting point

• Must evaluate how UDP traffic will react to a packet drop

• Right answer depends on the network

• Starting with IOS release 12.1.19EW1 and higher, the policer calculations can

include the 14 byte Ethernet header field and 4 byte FCS field when policing packets; this would be enabled using the global command: qos account

layer2 encapsulation length 18

• Releases prior to this do not include these fields; the policing rate and

burst parameters configured needed to deduct the layer 2 encapsulation length when determining the policing rate and burst, otherwise

“underpolicing” would result, particularly for smaller packet sizes in the 64

byte to 256 byte range

RST-3508 9805_05_2004_c2

115115115

Policed DSCP Mapping

Check if the Policed DSCP Table is Correctly Programmed

Cat4500(config)# qos map dscp policed 24 to dscp 16

Sup4#sh qos maps dscp policed Policed DSCP Mapping Table (dscp = d1d2) d1 : d2 0 1 2 3 4 5 6 7 8 9

------------------------------------0 : 00 01 02 03 04 05 06 07 08 09 1 : 10 11 12 13 14 15 16 17 18 19 2 : 20 21 22 23 16 25 26 27 28 29 3 : 30 31 32 33 34 35 36 37 38 39 4 : 40 41 42 43 44 45 46 47 48 49 5 : 50 51 52 53 54 55 56 57 58 59 6 : 60 61 62 63

When the Rate Is Exceeded DSCP of 24 Is Marked Down to 16

RST-3508 9805_05_2004_c2

116116116

Packet Transmit Queuing

Check DSCP to TX Queue Mappings

cat4500(config)# qos map dscp 50 to tx-queue 2

cat4500# sh qos maps dscp tx-queue

DSCP-TxQueue Mapping Table (dscp = d1d2)

d1 : d2 0 1 2 3 4 5 6 7 8 9

-------------------------------------

0 : 01 01 01 01 01 01 01 01 01 01

1 : 01 01 01 01 01 01 02 02 02 02

2 : 02 02 02 02 02 02 02 02 02 02

3 : 02 02 03 03 03 03 03 03 03 03

4 : 03 03 03 03 03 03 03 03 04 04

5 : 02 04 04 04 04 04 04 04 04 04

6 : 04 04 04 04

For DSCP of 50 TX Queue Is 2

RST-3508 9805_05_2004_c2

Input/Output Policer Rules Summary

If a Packet Encounters Both Input and Output Policy:

Ingress Policy

117117117

MarkMarkdownDropTransmitEgress Policy

MarkMarkdownDropTransmitTransmit

DropDropDropDropDrop

MarkdownMarkdownDropMarkdownMarkdown

RST-3508

The Most Severe Action Is Taken

9805_05_2004_c2

MarkMarkDropMarkMark

118118118

Dynamic Buffer Limiting (DBL)

• Congestion avoidance technique

• Flow based and maintains flow table per queue

• Operates by tracking buffer usage and credits

• If buffer usage exceeds dynamically computed

limit, DBL can either drop or set explicit congestion notification

• Implemented in Cisco IOS supervisor hardware with

line-rate performance

• The default DBL computation is very reliable; the

rule is not to tune the algorithm unless it is really required and the reason understood

RST-3508 9805_05_2004_c2

Dynamic Buffer Limiting

Drop One

Packet

BUFFERS

Aggressive

Buffer Limit

2 Pkts

CREDITS

Max Credits

15 (Default)

Aggressive

Credit

Limit 10

DBL

FLOW

119119119

Classified NAF

T0 T1 T2 T3 T4

NAF: Non Adaptive Flow

RST-3508 9805_05_2004_c2

Time

120120120

Enabling QoS and DBL on Cisco IOS Supervisors

The Information Is Applied per Port per Queue

Cat4500(config)#qos

Cat4500(config)#qos dbl

Cat4500# show qos dbl

DBL is enabled globally DBL flow includes vlan DBL flow includes layer4-ports DBL does not use ecn to indicate congestion DBL exceed-action probability: 15% DBL max credits: 15 DBL aggressive credit limit: 10 // NAF

threshold

DBL aggressive buffer limit: 2 packets // NAF’s

are limited

RST-3508 9805_05_2004_c2

121121121

Dynamic Buffer Limiting…Activated

C4506(config)# policy-map LAB-POLICY

C4506(config-pmap)# class UDP

C4506(config-pmap-c)# dbl

C4506(config-pmap)# class FTP

C4506(config-pmap-c)# dbl

RST-3508 9805_05_2004_c2

C4506# show policy

Policy Map LAB-POLICY

class FTP

class UDP

class WEB

class TELNET

set ip dscp 0

dbl

set ip dscp 0

dbl

set ip dscp 16

set ip dscp 48

122122122

DBL Troubleshooting: Is It Working?

Cat4500#sh int gig4/1 count detail

( truncated )

Port Tx-Bytes-Queue-1 Tx-Bytes-Queue-2 Tx-Bytes-Queue-3 Tx-Bytes-Queue-4 Gi4/1 11114432 0 64000 0

Port Tx-Drops-Queue-1 Tx-Drops-Queue-2 Tx-Drops-Queue-3 Tx-Drops-Queue-4 Gi4/1 99925 0 0 0

Port Dbl-Drops-Queue-1 Dbl-Drops-Queue-2 Dbl-Drops-Queue-3 Dbl-Drops-Queue

Gi4/1 73425 0 0

DBL Drops

0

RST-3508 9805_05_2004_c2

Associated Sessions

• RST-3511: Troubleshooting LAN Protocols

• RST-3509: Catalyst 6500 Troubleshooting

• RST-3507: Catalyst 2900 and Catalyst 3500

Troubleshooting

123123123

RST-3508 9805_05_2004_c2

124124124

For More Information:

• Understanding and Configuring QoS on Catalyst 4500 Series Switches

http://www.cisco.com/en/US/partner/products/hw/switches/ps4324/products_regional_sales_pr omotion09186a00801fcabd.html

• Understanding and Configuring IP Multicast on Catalyst 4000 Series

Switches

http://www.cisco.com/univercd/cc/td/doc/product/lan/cat4000/12_2_18EW/config/mcastmls.htm

• Security Best Practices on Catalyst 4500 Series Switches

http://www.cisco.com/en/US/partner/products/hw/switches/ps4324/products_regional_sales_pr omotion09186a00801fcabd.html

• Catalyst 4500 Power over Ethernet Capabilities

http://www.cisco.com/en/US/partner/products/hw/switches/ps4324/products_regional_sales_pr omotion09186a00801fcabd.html

• Hardware Troubleshooting for Catalyst 4000/4912G/2980G/2948G Series

Switches

http://www.cisco.com/warp/customer/473/121.html

• Troubleshooting Hardware and Related Issues on Catalyst 4000 and 4500

Supervisor III and IV

http://www.cisco.com/warp/customer/473/165.html

• Catalyst 4000 Series Documnetation

http://www.cisco.com/univercd/cc/td/doc/product/lan/cat4000/index.htm

RST-3508 9805_05_2004_c2

125125125

Troubleshooting Support for Cisco Catalyst 4000 Series Switches: Cisco Technical Support Website

www.cisco.com/techsupport

Troubleshooting Section Includes:

• Known problems

(e.g., release notes, field notices, security advisories)

• Troubleshooting

resources for common error messages, CPU utilization, etc., and troubleshooting tools (e.g., TAC case collection)

RST-3508 9805_05_2004_c2

Troubleshooting

Resources

126126126

Troubleshooting Support for Cisco Catalyst 4000 Series Switches: Cisco Technical Support Website

www.cisco.com/techsupport

Troubleshooting Section Includes:

• Known problems

(e.g., release notes, field notices, security advisories)

• Troubleshooting

resources for common error messages, CPU utilization, etc., and troubleshooting tools (e.g., TAC case collection)

Troubleshoot

ing

Resources

RST-3508 9805_05_2004_c2

Q AND A

127127127

RST-3508 9805_05_2004_c2

128128128

Cisco WS-C4507R, Catalyst 4000, Catalyst 4500 Troubleshooting Manual

Specifications and Main Features

Frequently Asked Questions

User Manual