Is it normal for CPU to spike to 100% on a 4506?

[biwood]

Our 4506 is our core switch as well as the bridge in our network. Every port is connected and utilized. This branch is an R&D center so it's pretty busy. When I do a "sh cpu proc hist" I get:

0810190025299362978996509996499894482289792096
100 * ** * * * * **
90 ** ** * ** * * **
80 ** ***** * ** ** * **
70 *** ***** ******** * **
60 * * *********** * ******** *****
50 * *************** * ***************
40 *******************************************#**
30 *******************************************#**
20 **********##*******##***********#*##*#*****##*
10 ##############################################
0....5....1....1....2....2....3....3....4....4.
0 5 0 5 0 5 0 5

As you can see the average only jumps to 40% but the max is up there quite frequently. We have been noticing some network slowness from time to time but we're not sure if the switch or not since port forwarding occurs in hardware.
Is it normal for a heavy utilized switch to spike up to 100% that frequently?

 

[sirlollypop]

I would say no. You have something wrong! I dont use 4500 only 6509 but I have never seen that on one of ours!

When you say every port what blades do you have in the Cat?

 

[vipergg]

That is pretty high even for a 4500 switch . This might help to look for it.

http://cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml

 

[vipergg]

Newer code versions have a neat little feature that not too many people know about .

Tool 2: In-Built CPU Sniffer—Cisco IOS Software Release 12.2(20)EW and Later

The Catalyst 4500 provides an in-built CPU sniffer and decoder to quickly identify the traffic that hits the CPU. You can enable this facility with the debug command, as the example in this section shows. This features implements a circular buffer that can retain 1024 packets at a time. As new packets arrive, they overwrite the older packets. This feature is safe to use when you troubleshoot high CPU utilization issues.

Switch#debug platform packet all receive buffer
platform packet debugging is on
Switch#show platform cpu packet buffered
Total Received Packets Buffered: 36
-------------------------------------
Index 0:
7 days 23:6:32:37214 - RxVlan: 99, RxPort: Gi4/48
Priority: Crucial, Tag: Dot1Q Tag, Event: Control Packet, Flags: 0x40, Size: 68
Eth: Src 00-0F-F7-AC-EE-4F Dst 01-00-0C-CC-CC-CD Type/Len 0x0032
Remaining data:
0: 0xAA 0xAA 0x3 0x0 0x0 0xC 0x1 0xB 0x0 0x0
10: 0x0 0x0 0x0 0x80 0x0 0x0 0x2 0x16 0x63 0x28
20: 0x62 0x0 0x0 0x0 0x0 0x80 0x0 0x0 0x2 0x16
30: 0x63 0x28 0x62 0x80 0xF0 0x0 0x0 0x14 0x0 0x2
40: 0x0 0xF 0x0 0x0 0x0 0x0 0x0 0x2 0x0 0x63
Index 1:
7 days 23:6:33:180863 - RxVlan: 1, RxPort: Gi4/48
Priority: Crucial, Tag: Dot1Q Tag, Event: Control Packet, Flags: 0x40, Size: 68
Eth: Src 00-0F-F7-AC-EE-4F Dst 01-00-0C-CC-CC-CD Type/Len 0x0032
Remaining data:
0: 0xAA 0xAA 0x3 0x0 0x0 0xC 0x1 0xB 0x0 0x0
10: 0x0 0x0 0x0 0x80 0x0 0x0 0x2 0x16 0x63 0x28
20: 0x62 0x0 0x0 0x0 0x0 0x80 0x0 0x0 0x2 0x16
30: 0x63 0x28 0x62 0x80 0xF0 0x0 0x0 0x14 0x0 0x2
40: 0x0 0xF 0x0 0x0 0x0 0x0 0x0 0x2 0x0 0x63

Note: The CPU utilization when you issue a debug command is always almost 100%. It is normal to have high CPU utilization when you issue a debug command.

 

[biwood]

SirPop,
We have the Gig blades. I'm not sure exactly which ones w/o logging in. I think they are at level 4.

Vipergg,
Yeah, I read that doc. I didn't want to go through the whole process if it was considered normal. I also read about those tools in the newer IOS. The problem here is that we just took over management of this switch which is in Beijing, so we can't be too invasive and all the high CPU traffic is at like 3a.m. EST.

This is one of those things where the boss is replacing the hardware anyway but he's afraid if we don't figure out the problem will have the same issue with the new hardware. I'm not sure if it's worth it to upgrade the IOS at this point so it looks like I'm going to have logon at 3 a.m. and use the CPU troubleshooting doc from Cisco.

 

[BuckWeet]

is multicast running?? the switch cpu could be processing multicast traffic if improperly configured..

 

[helpdeskdan]

Have you setup span and looked at the traffic in wireshark? Not the best tool for the job, but simple and free.

CPU Sniffer?! News to me - Where does one read about this stuff? I apologize for posting off topic, but I am wondering if there is a Cisco website or magazine or the like which I should be reading because I missed that announcement. Again, sorry for posting off the subject.

 

[biwood]

HDD,
There's detail on the CPU sniffer in the "cat4500_high_cpu.pdf" doc on Cisco's site. It's on page 30.

 

[helpdeskdan]

Thanks