High CPU usage/sendmail problem?

[jamisi]

I have a problem that hopefully one of you linux gurus can figure out...

I have had a linux server running for over 6 months now with a minimal load. It has a few sites on it, which are being used by a few folks and are being developed. I have a graphical stats utility on the server called HotSANIC, similar to MRTG. Anyway, one of the developers noticed the server running a bit slow so I checked the HotSANIC stats yesterday and noticed something very odd. Starting about a week ago the CPU usage started increasing steadily, and now it's running at around 60% consistently. Also, disk space is being filled up at a steady rate, consuming nearly 20GB over the last week. Processes seem to have increased from an average of about 100 to about 150, and server load is up also. Network activity and memory usage seem to be normal, however. I investigated a bit (I'm not super experienced in linux system administration, I'm a beginner-intermediate level I guess) and according to top, the CPU usage is by the system, not user processes. I also noticed a lot of sendmail processes, but they aren't using much CPU, so I looked at the maillogs and they seem to be where a lot of the disk space is being consumed, but not all of it. They're huge, the current one is over 5GB for the last 3 days, and full of entries like this:

Dec 18 04:04:38 home sendmail[13607]: jBG2G0uU022356: to=crystal, delay=2+07:48:38, xdelay=00:00:00, mailer=local, pri=4450364, dsn=4.0.0, stat=Deferred: local mailer (/usr/bin/procmail) exited with EX_TEMPFAIL

Anyone have any idea what could be going on, what could have caused this, and what I can do to fix it? Here is the link to our HotSANIC stats so you can see what I'm talking about:

http://home.netword.com/stats/

You'll notice that the server was up for about 180 days before this happened, running just fine. I rebooted yesterday to see if that fixed the problem, but it didn't.

Thanks in advance.

 

[jamisi]

edit: the sendmail logs are not 5GB for the last 3 days, they are 500MB for the last 3 days. Still huge, but not THAT huge...so I'm not sure what is taking up all the disk space at this point...

Thanks.

 

[pansophic]

Try doing a "du -h > day1" from / as root. The try again the next day using day2 as the file name. Then diff the two files. It should be readily apparent where the disk space is going.

If it isn't apparent, then you may want to download chkrootkit and compile it as a statically linked binary on a system that you know is not compromised. Then run it on this one.

Who knows, you could have been hacked, but it is probably just an application that is running rampant.

If you have a windowing environment, you might want to try gtop or ksysguard so that you can sort by user and system processes readily. You may be able to spot a runaway process there.


pansophic

 

[Annihilannic]

You can also use dispus to find out where all that space is being used.

I think they key thing is to find out why /usr/bin/procmail is exiting with that error. If you run mailq does it show a gigantic mail queue? Perhaps that is what your server is spending all its time managing.

Annihilannic.

 

[jamisi]

Thanks for the replies pansophic and Annihilannic, you guys were very helpful. It turns out that Annihilannic was right, it was a mail queue issue. There was a catch-all set on the domain to dump all invalid mail to an admin user, and the domain suddenly got slammed with a ton of spam which filled up the 2GB limit on the admin user's mailbox (wasn't a box that the admin was checking periodically...) and backed up into the queue. The queue grew to over 117,000+ emails, which bogged down the server.

Got it all cleared out now, and got rid of that friggin' catch-all alias...Thanks again guys!

J