Interpreting CPU Usage - 3640 Router 1

[hinesjrh]

I am using portions of Solarwinds to monitor my network. One tool I use is the Router CPU Load, which on occassion shows my 3640 router hitting near CPU capacity (>95%). However, when I do a 'show proc cpu' on the router I don't see anything that high. Is my Solarwinds tool misleading, or do I not know how to read the output from the router command? Any help is sure appreciated!



Corporate#show proc cpu
CPU utilization for five seconds: 42%/40%; one minute: 40%; five minutes: 42%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 0 1 0 0.00% 0.00% 0.00% 0 Chunk Manager
2 21704 8953486 2 0.00% 0.00% 0.00% 0 Load Meter
3 116 110 1054 0.49% 0.05% 0.01% 130 Virtual Exec
4 12700768 4549717 2791 0.00% 0.01% 0.00% 0 Check heaps
5 146204 15810 9247 0.00% 0.00% 0.00% 0 Pool Manager
6 0 2 0 0.00% 0.00% 0.00% 0 Timers
7 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun
8 45048 8951776 5 0.00% 0.00% 0.00% 0 ALARM_TRIGGER_SC
9 0 1 0 0.00% 0.00% 0.00% 0 OIR Handler
10 3400 1492274 2 0.00% 0.00% 0.00% 0 Environmental mo
11 135062024 188140553 717 0.65% 0.36% 0.32% 0 ARP Input
12 26212 2238400 11 0.00% 0.00% 0.00% 0 HC Counter Timer
13 0 3 0 0.00% 0.00% 0.00% 0 DDR Timers
14 0 2 0 0.00% 0.00% 0.00% 0 Dialer event
15 0 2 0 0.00% 0.00% 0.00% 0 Entity MIB API
16 0 1 0 0.00% 0.00% 0.00% 0 SERIAL A'detect
17 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd
18 277936 10682470 26 0.00% 0.00% 0.00% 0 Net Background
19 37044 2176413 17 0.00% 0.00% 0.00% 0 Logger
20 161712 44761193 3 0.00% 0.00% 0.00% 0 TTY Background
21 286160 44786876 6 0.00% 0.00% 0.00% 0 Per-Second Jobs
22 0 2 0 0.00% 0.00% 0.00% 0 Hawkeye Backgrou
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
23 18432416 116142878 158 0.08% 0.04% 0.01% 0 Net Input
24 54264 8953492 6 0.00% 0.00% 0.00% 0 Compute load avg
25 8467716 776006 10911 0.00% 0.00% 0.00% 0 Per-minute Jobs
26 2300 1492275 1 0.00% 0.00% 0.00% 0 Call Management
27 0 2 0 0.00% 0.00% 0.00% 0 AAA Dictionary R
28 319506284 334896589 954 0.57% 0.53% 0.52% 0 IP Input
29 412312 5765233 71 0.00% 0.00% 0.00% 0 CDP Protocol
30 0 1 0 0.00% 0.00% 0.00% 0 X.25 Encaps Mana
31 0 1 0 0.00% 0.00% 0.00% 0 Asy FS Helper
32 0 1 0 0.00% 0.00% 0.00% 0 PPP IP Add Route
33 5637360 759465 7422 0.00% 0.00% 0.00% 0 IP Background
34 170476 746237 228 0.00% 0.00% 0.00% 0 IP RIB Update
35 331548 750327 441 0.00% 0.00% 0.00% 0 Adj Manager
36 13068 99684 131 0.00% 0.00% 0.00% 0 TCP Timer
37 92708 81096 1143 0.00% 0.00% 0.00% 0 TCP Protocols
38 0 1 0 0.00% 0.00% 0.00% 0 Probe Input
39 156 888 175 0.00% 0.00% 0.00% 0 RARP Input
40 0 1 0 0.00% 0.00% 0.00% 0 HTTP Timer
41 0 1 0 0.00% 0.00% 0.00% 0 Socket Timers
42 1804384 1994309 904 0.00% 0.01% 0.00% 0 DHCPD Receive
43 36504 746019 48 0.00% 0.00% 0.00% 0 IP Cache Ager
44 0 1 0 0.00% 0.00% 0.00% 0 COPS
45 0 1 0 0.00% 0.00% 0.00% 0 PAD InCall
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
46 0 2 0 0.00% 0.00% 0.00% 0 X.25 Background
48 0 1 0 0.00% 0.00% 0.00% 0 Syslog Traps
49 13047716 66811849 195 0.16% 0.13% 0.13% 0 CEF process
50 0 4 0 0.00% 0.00% 0.00% 0 SNMP Timers
51 10452844 14662605 712 0.08% 0.00% 0.00% 0 IP SNMP
52 4169560 7880382 529 0.00% 0.00% 0.00% 0 PDU DISPATCHER
53 18873336 7918086 2383 0.00% 0.00% 0.00% 0 SNMP ENGINE
54 560 24 23333 0.00% 0.00% 0.00% 0 SNMP ConfCopyPro
55 0 1 0 0.00% 0.00% 0.00% 0 SNMP Traps
56 121260 44872025 2 0.00% 0.00% 0.00% 0 NTP
57 75192 1552551 48 0.00% 0.00% 0.00% 0 CEF Scanner
58 716 373083 1 0.00% 0.00% 0.00% 0 DHCPD Timer
59 268896 12682308 21 0.00% 0.00% 0.00% 0 DHCPD Database
Corporate#

 

[hinesjrh]

Here's some additional CPU usage information from 'show tech' on my router:

------------------ show process cpu history ------------------


4444333333333333333333333333355555444444444433333333333333
1111999998888899999555558888833333555555555577777777776666
100
90
80
70
60
50 ***************
40 **********************************************************
30 **********************************************************
20 **********************************************************
10 **********************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5

CPU% per second (last 60 seconds)


5444444455854444444444565593333334433333333334443443444444
3223322302222665765454317378556672277798996882228039122321
100 *
90 *
80 * *
70 * *
60 * ** #
50 * *##* ****** * #####
40 #*#########################*******#**********##***###*****
30 ##########################################################
20 ##########################################################
10 ##########################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5

CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%

1
8899463111444113142488099999843122423113132488999888553122312223333377
9846499867250978885108064035979812208754643405990278325801436084052608
100 * ** * **
90 **** ******** **** **
80 **** ********* ******** *
70 **** * ********* ******** **
60 **** * ********* ******** **
50 **** * * * ********** ********** **
40 ***#*** *** * * ****##****** * * ***####***** * ***
30 ######* *** * ****#########* * * * * **#########* * * ********
20 #######***#****#*#*############***#********############**** *********#
10 ######################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0

CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%

 

[KiscoKid]

The graphs you've provided do show a couple of peaks in excess of 95%.

For example, on the last 60 mins graph, it goes north of 95 once. On the last 72 hours however, there's quite a few 90+ readings there.

In case you're not aware the top two rows, read vertically, display the highest percentage of CPU utilization recorded during the increment. Those figures represent the max CPU measured in that period.

The # symbol represents the average during the same period.

For example, I see 2 successive intervals on the 72 hour graph where the CPU maxed out at 99%. The average however during those intervals was around 40% however.

 

[hinesjrh]

Thanks for the feedback KiscoKid! So my Solarwinds tool is telling a true story.

Now, what do you suggest to get to the bottom of what might be causing this? I assume that going above 90% CPU usage even on occassion is not a good thing. Can more memory be added (would that even help here), or is it time for a bigger router?

 

[KiscoKid]

In my opinion, I wouldn't be too concerned when my devices occasionally hit a high CPU - this sometimes is just an indication of higher than normal bursty traffic which is normal network behaviour. Sustained and continuous high utilisation is a bad thing however esp when it results in user traffic being affected.

Troubleshooting CPU issues is not always easy but ultimately you need to identify if the issue is with the processes running on the box (e.g. IP input) or with interrupts (e.g. NAT uses CPU interrupts). In case you're not aware, the reason there is 2 CPU Utilisations reported in the 'show cpu proc' command is that one figure reports the total CPU utilisation whilst the 2nd one reports the CPU utilisation due to interrupts. In the shapshot you presented, it shows 42%/40% which tells me 42% is total CPU util and 40% CPU utilisation is due to interrupts. The list of processes listed below account for less than 2% of the total utilisation. So it's likely when it peaks over 90% this is also probably due to interrupts.

Cisco have released a number of documents concerning troubleshooting CPU issues. I've posted them here if you wish to peruse them but ultimately I'm not too sure your router is suffering at this stage.

After all, even though I saw the router maxing out at 99%, it's average was around 40% which is more than acceptable in my opinion.

http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a00800a70f2.shtml

http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af0.shtml

Hope this helps

 

[tech1994]

I also use Solarwinds (Orion) and monitor CPU usage among other things.
I think posting a config and some additional information may be useful as to determine where your high CPU use is coming from.

----------------------

http://www.ciscohq.com

 

[hinesjrh]

tech1994: Here's the config and interface details. FYI, I have just seen the CPU hit 90% percent in the past couple of months. The 12 Mb (partial DS3) does hit 100% usage for small amounts of time, but averages <25% input and <40% output during busy business days (less at night).

Corp#show config
Using 7659 out of 129016 bytes
!
! Last configuration change at 23:55:00 eastern Thu Oct 13 2005
! NVRAM config last updated at 23:58:31 eastern Thu Oct 13 2005
!
version 12.2
service timestamps debug uptime
service timestamps log uptime
service password-encryption
service udp-small-servers
service tcp-small-servers
!
hostname Corporate
!
boot system flash:c3640-i-mz.122-17.bin
logging buffered 4096 debugging
enable secret 5 $1$TKEC$BcwafKIfamNZC5yUseGto.
enable password 7 060807204F411B091718071A0916
!
clock timezone eastern -5
ip subnet-zero
no ip icmp rate-limit unreachable
ip cef
!
!
ip domain-name nhacentral.com
!
!
!
!
interface FastEthernet0/0
ip address 10.1.0.3 255.255.252.0
speed 100
full-duplex
!
interface FastEthernet0/1
no ip address
duplex auto
speed auto
!
interface Hssi2/0
description Qwest DS3 PRN Cir# DS3-7356180
ip address 69.115.164.134 255.255.255.252
ip access-group 104 in
ip access-group 100 out
fair-queue
serial restart-delay 0
!
ip classless
ip route 0.0.0.0 0.0.0.0 10.1.0.2
ip route 10.2.0.0 255.255.255.0 65.119.164.133
ip route 10.3.0.0 255.255.255.0 65.119.164.133
ip route 10.4.0.0 255.255.255.0 65.119.164.133
ip route 10.5.0.0 255.255.255.0 65.119.164.133
ip route 10.6.0.0 255.255.255.0 65.119.164.133
ip route 10.7.0.0 255.255.255.0 65.119.164.133
ip route 10.8.0.0 255.255.255.0 65.119.164.133
ip route 10.9.0.0 255.255.255.0 65.119.164.133
ip route 10.11.0.0 255.255.255.0 65.119.164.133
ip route 10.14.0.0 255.255.255.0 65.119.164.133
ip route 10.15.0.0 255.255.255.0 65.119.164.133
ip route 10.16.0.0 255.255.255.0 65.119.164.133
ip route 10.18.0.0 255.255.255.0 65.119.164.133
ip route 10.19.0.0 255.255.255.0 65.119.164.133
ip route 10.20.0.0 255.255.255.0 65.119.164.133
ip route 10.21.0.0 255.255.255.0 65.119.164.133
ip route 10.22.0.0 255.255.255.0 65.119.164.133
ip route 10.23.0.0 255.255.255.0 65.119.164.133
ip route 10.24.0.0 255.255.255.0 65.119.164.133
ip route 10.25.0.0 255.255.255.0 65.119.164.133
ip route 10.26.0.0 255.255.255.0 65.119.164.133
ip route 10.27.0.0 255.255.255.0 65.119.164.133
ip route 10.36.0.0 255.255.255.0 65.119.164.133
ip route 10.37.0.0 255.255.255.0 65.119.164.133
ip route 10.38.0.0 255.255.255.0 65.119.164.133
ip route 10.39.0.0 255.255.255.0 65.119.164.133
ip route 10.40.0.0 255.255.255.0 65.119.164.133
ip route 10.41.0.0 255.255.255.0 65.119.164.133
ip route 10.42.0.0 255.255.255.0 65.119.164.133
ip route 10.43.0.0 255.255.255.0 65.119.164.133
ip route 10.44.0.0 255.255.255.0 65.119.164.133
ip route 10.45.0.0 255.255.255.0 65.119.164.133
ip route 10.46.0.0 255.255.255.0 65.119.164.133
ip route 10.47.0.0 255.255.255.0 65.119.164.133
ip route 10.48.0.0 255.255.255.0 65.119.164.133
ip route 10.49.0.0 255.255.255.0 65.119.164.133
ip route 10.50.0.0 255.255.255.0 65.119.164.133
ip route 10.51.0.0 255.255.255.0 65.119.164.133
ip route 10.52.0.0 255.255.255.0 65.119.164.133
ip route 10.53.0.0 255.255.255.0 65.119.164.133
ip route 10.54.0.0 255.255.255.0 65.119.164.133
ip route 10.55.0.0 255.255.255.0 65.119.164.133
ip route 10.56.0.0 255.255.255.0 65.119.164.133
ip route 10.57.0.0 255.255.255.0 65.119.164.133
ip route 10.58.0.0 255.255.255.0 65.119.164.133
ip route 10.59.0.0 255.255.255.0 65.119.164.133
ip route 10.60.0.0 255.255.255.0 65.119.164.133
ip route 10.61.0.0 255.255.255.0 65.119.164.133
ip route 10.64.0.0 255.255.255.0 65.119.164.133
ip route 10.65.0.0 255.255.255.0 65.119.164.133
ip route 10.66.0.0 255.255.255.0 65.119.164.133
ip route 10.71.0.0 255.255.255.0 65.119.164.133
ip route 10.251.0.0 255.255.0.0 10.1.0.6
ip route 10.252.0.0 255.255.0.0 10.1.0.6
ip http server
!
logging facility local4
logging 10.1.0.121
access-list 100 permit tcp host 10.1.1.119 any
access-list 100 permit udp host 10.1.1.119 any
access-list 100 permit udp any any eq 2344
access-list 100 permit udp any any eq 16962
access-list 100 deny udp any any eq 8998 log
access-list 100 deny udp any any eq tftp log
access-list 100 permit tcp host 10.1.0.150 any eq 135
access-list 100 permit tcp host 10.1.0.127 any eq 135
access-list 100 permit tcp host 10.1.0.152 any eq 135
access-list 100 permit tcp host 10.1.0.154 any eq 135
access-list 100 permit tcp host 10.1.2.24 any eq 135
access-list 100 permit tcp host 10.253.0.126 any eq 135
access-list 100 permit tcp host 10.253.0.66 any eq 135
access-list 100 permit tcp host 10.1.0.66 any eq 135
access-list 100 permit tcp host 10.1.0.65 any eq 135
access-list 100 permit tcp host 10.1.0.156 any eq 135
access-list 100 permit tcp host 10.1.0.134 any eq 135
access-list 100 permit tcp 10.0.0.16 0.255.0.0 any eq 135
access-list 100 permit tcp host 10.1.0.120 any eq 135
access-list 100 deny tcp any any eq 135 log
access-list 100 permit icmp host 10.1.0.3 any
access-list 100 permit icmp host 205.171.3.169 any
access-list 100 permit icmp host 65.119.164.133 any
access-list 100 permit icmp host 10.1.1.119 any
access-list 100 permit icmp host 10.1.0.134 any
access-list 100 permit icmp host 10.253.0.126 any
access-list 100 permit icmp host 10.1.0.150 any
access-list 100 permit icmp host 10.1.0.152 any
access-list 100 permit icmp host 10.1.0.154 any
access-list 100 permit icmp host 10.253.0.66 any
access-list 100 permit icmp host 10.1.0.66 any
access-list 100 permit icmp host 10.1.0.65 any
access-list 100 permit icmp 10.0.0.16 0.255.0.0 any
access-list 100 permit icmp 10.0.0.17 0.255.0.0 any
access-list 100 permit ip any any
access-list 104 permit tcp any host 10.1.1.119
access-list 104 permit udp any host 10.1.1.119
access-list 104 permit udp any any eq 2344
access-list 104 permit udp any any eq 16962
access-list 104 deny udp any any eq 8998 log
access-list 104 deny udp any any eq tftp log
access-list 104 permit tcp any host 10.1.0.150 eq 135
access-list 104 permit tcp any host 10.1.0.152 eq 135
access-list 104 permit tcp any host 10.1.0.154 eq 135
access-list 104 permit tcp any host 10.1.2.24 eq 135
access-list 104 permit tcp any host 10.253.0.126 eq 135
access-list 104 permit tcp any host 10.253.0.66 eq 135
access-list 104 permit tcp any host 10.1.0.66 eq 135
access-list 104 permit tcp any host 10.1.0.65 eq 135
access-list 104 permit tcp any host 10.1.0.156 eq 135
access-list 104 permit tcp any host 10.1.0.134 eq 135
access-list 104 permit tcp any 10.0.0.16 0.255.0.0 eq 135
access-list 104 permit tcp any host 10.1.0.120 eq 135
access-list 104 deny tcp any any eq 135 log
access-list 104 permit icmp any host 10.1.1.119
access-list 104 permit icmp any host 10.1.0.134
access-list 104 permit icmp any host 10.253.0.126
access-list 104 permit icmp any host 10.1.0.150
access-list 104 permit icmp any host 10.1.0.152
access-list 104 permit icmp any host 10.1.0.154
access-list 104 permit icmp any host 10.253.0.66
access-list 104 permit icmp any host 10.1.0.66
access-list 104 permit icmp any host 10.1.0.65
access-list 104 permit icmp any 10.0.0.16 0.255.0.0
access-list 104 permit icmp any 10.0.0.17 0.255.0.0
access-list 104 deny udp any any eq 995 log
access-list 104 deny udp any any eq 996 log
access-list 104 deny udp any any eq 997 log
access-list 104 deny udp any any eq 998 log
access-list 104 deny udp any any eq 999 log
access-list 104 permit ip any any
access-list 2000 permit ip any any
access-list 2000 permit icmp any any
snmp-server community haveaniceday RW
snmp-server enable traps tty
snmp-server host 10.1.0.121 public
!
line con 0
password 7 01455Q07530A1408245E
login
transport preferred none
line aux 0
password 7 1544530S0C2B39232U21
login
line vty 0 4
password 7 13534B11030D162D2T36
login
transport preferred none
line vty 5
login
transport preferred none
!
ntp clock-period 17180041
ntp server 130.126.24.53
end

Corp#



Corp#show interface
FastEthernet0/0 is up, line protocol is up
Hardware is AmdFE, address is 0001.42a5.5ee1 (bia 0001.42a5.5ee1)
Internet address is 10.1.0.3/22
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 4/255, rxload 12/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 3d01h
Input queue: 0/75/3725/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 4917000 bits/sec, 946 packets/sec
5 minute output rate 1861000 bits/sec, 881 packets/sec
204395399 packets input, 2079539879 bytes
Received 3745687 broadcasts, 0 runts, 0 giants, 0 throttles
43993 input errors, 0 CRC, 0 frame, 43993 overrun, 0 ignored
0 watchdog
0 input packets with dribble condition detected
195099010 packets output, 2500981187 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
FastEthernet0/1 is up, line protocol is down
Hardware is AmdFE, address is 0001.42a5.5ee2 (bia 0001.42a5.5ee2)
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Auto-duplex, Auto Speed, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:03, output hang never
Last clearing of "show interface" counters 3d01h
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog
0 input packets with dribble condition detected
26369 packets output, 1582140 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
Hssi2/0 is up, line protocol is up
Hardware is M1T-HSSI-B
Description: Qwest DS3 PRN Cir# DS3-7356180
Internet address is 69.115.164.134/30
MTU 4470 bytes, BW 45045 Kbit, DLY 200 usec,
reliability 255/255, txload 26/255, rxload 9/255
Encapsulation HDLC, crc 16, loopback not set
Keepalive set (10 sec)
Restart-Delay is 0 secs
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 3d01h
Input queue: 0/75/357/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: weighted fair
Output queue: 0/1000/64/0 (size/max total/threshold/drops)
Conversations 0/112/256 (active/max active/max total)
Reserved Conversations 0/0 (allocated/max allocated)
Available Bandwidth 33783 kilobits/sec
5 minute input rate 1726000 bits/sec, 833 packets/sec
5 minute output rate 4730000 bits/sec, 885 packets/sec
188051052 packets input, 3303043836 bytes, 151 no buffer
Received 26388 broadcasts, 0 runts, 0 giants, 0 throttles
0 parity
11952 input errors, 0 CRC, 0 frame, 11952 overrun, 1 ignored, 0 abort
193851708 packets output, 1485022713 bytes, 20005 underruns
0 output errors, 0 applique, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions LC=down CA=up TM=down LB=down TA=up LA=down

Corp#

 

[tech1994]

1st and formost get a moderator to edit your post, you just gave away your whole config, public ips, pass's the whole ball of wax. Get that edited ASAP.

At a quick glance and not knowing your whole network layout and goals.
ACL's - Have you looked into improvements of your list, could be a huge performance increase.
Routing - Have you considered using a routing protocol, maybe EIGRP vs static's?

----------------------

http://www.ciscohq.com

 

[hinesjrh]

No need to worry tech1994, I changed any usable info. in the config. before posting! Find and replace comes in handy there.

Good point on the ACL's. All those existed when I started here and I have not had a need (yet) to change or look further into them. I'll put that on the to-do list.