How to rotate syslog files

[zhenning]

We use msyslogd and it is now configured to rotate after 14 log files at /etc/logrotate.d/syslog_cisco_fwsm:

/var/log/cisco_fwsm.log {
size=200M
rotate 14
postrotate
/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
endscript
}

I changed to rotate 70 and restarted the box. But it did not rotate after the log file went over 200M. Then the msyslogd stopped.

Any ideas why it did not rotate properly?

Thanks!
Zhenning

 

[ZaSter]

zhenning,

Can you run it in debug/test-mode? What do you get when you run:

logrotate -d -f /etc/logrotate.d/syslog_cisco_fwsm


ZaSter

 

[zhenning]

I got:

[root@scylla logrotate.d]# /usr/sbin/logrotate -d -f /etc/logrotate.d/syslog_cisco_fwsm
reading config file /etc/logrotate.d/syslog_cisco_fwsm
reading config info for /var/log/cisco_fwsm.log

Handling 1 logs

rotating pattern: /var/log/cisco_fwsm.log forced from command line (14 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/cisco_fwsm.log
log needs rotating
rotating log /var/log/cisco_fwsm.log, log->rotateCount is 14
renaming /var/log/cisco_fwsm.log.14 to /var/log/cisco_fwsm.log.15 (rotatecount 14, logstart 1, i 14),
renaming /var/log/cisco_fwsm.log.13 to /var/log/cisco_fwsm.log.14 (rotatecount 14, logstart 1, i 13),
renaming /var/log/cisco_fwsm.log.12 to /var/log/cisco_fwsm.log.13 (rotatecount 14, logstart 1, i 12),
renaming /var/log/cisco_fwsm.log.11 to /var/log/cisco_fwsm.log.12 (rotatecount 14, logstart 1, i 11),
renaming /var/log/cisco_fwsm.log.10 to /var/log/cisco_fwsm.log.11 (rotatecount 14, logstart 1, i 10),
renaming /var/log/cisco_fwsm.log.9 to /var/log/cisco_fwsm.log.10 (rotatecount 14, logstart 1, i 9),
renaming /var/log/cisco_fwsm.log.8 to /var/log/cisco_fwsm.log.9 (rotatecount 14, logstart 1, i 8),
renaming /var/log/cisco_fwsm.log.7 to /var/log/cisco_fwsm.log.8 (rotatecount 14, logstart 1, i 7),
renaming /var/log/cisco_fwsm.log.6 to /var/log/cisco_fwsm.log.7 (rotatecount 14, logstart 1, i 6),
renaming /var/log/cisco_fwsm.log.5 to /var/log/cisco_fwsm.log.6 (rotatecount 14, logstart 1, i 5),
renaming /var/log/cisco_fwsm.log.4 to /var/log/cisco_fwsm.log.5 (rotatecount 14, logstart 1, i 4),
renaming /var/log/cisco_fwsm.log.3 to /var/log/cisco_fwsm.log.4 (rotatecount 14, logstart 1, i 3),
renaming /var/log/cisco_fwsm.log.2 to /var/log/cisco_fwsm.log.3 (rotatecount 14, logstart 1, i 2),
renaming /var/log/cisco_fwsm.log.1 to /var/log/cisco_fwsm.log.2 (rotatecount 14, logstart 1, i 1),
renaming /var/log/cisco_fwsm.log.0 to /var/log/cisco_fwsm.log.1 (rotatecount 14, logstart 1, i 0),
renaming /var/log/cisco_fwsm.log to /var/log/cisco_fwsm.log.1
running postrotate script
running script with arg /var/log/cisco_fwsm.log: "
/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
"
removing old log /var/log/cisco_fwsm.log.15

What could be wrong?

Thanks!
Zhenning

 

[ZaSter]

Zhenning,

So, what happens when you run it in actual-mode with verbose option?:

logrotate -v -f /etc/logrotate.d/syslog_cisco_fwsm

Does the /var/log partition have 200MB it can spare?

--
ZaSter

 

[zhenning]

The cisco_fwsm.log was configured to rotate every hour but not it stopped to rotate every hour. It went all the way to 200M then msyslogd will stop. Where is the configuration to rotate every hour?
Thanks!
Zhenning

 

[zhenning]

[root@scylla logrotate.d]# /usr/sbin/logrotate -v -f /etc/logrotate.d/syslog_cisco_fwsm
reading config file /etc/logrotate.d/syslog_cisco_fwsm
reading config info for /var/log/cisco_fwsm.log

Handling 1 logs

rotating pattern: /var/log/cisco_fwsm.log forced from command line (14 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/cisco_fwsm.log
log needs rotating
rotating log /var/log/cisco_fwsm.log, log->rotateCount is 14
renaming /var/log/cisco_fwsm.log.14 to /var/log/cisco_fwsm.log.15 (rotatecount 14, logstart 1, i 14),
old log /var/log/cisco_fwsm.log.14 does not exist
renaming /var/log/cisco_fwsm.log.13 to /var/log/cisco_fwsm.log.14 (rotatecount 14, logstart 1, i 13),
old log /var/log/cisco_fwsm.log.13 does not exist
renaming /var/log/cisco_fwsm.log.12 to /var/log/cisco_fwsm.log.13 (rotatecount 14, logstart 1, i 12),
old log /var/log/cisco_fwsm.log.12 does not exist
renaming /var/log/cisco_fwsm.log.11 to /var/log/cisco_fwsm.log.12 (rotatecount 14, logstart 1, i 11),
old log /var/log/cisco_fwsm.log.11 does not exist
renaming /var/log/cisco_fwsm.log.10 to /var/log/cisco_fwsm.log.11 (rotatecount 14, logstart 1, i 10),
old log /var/log/cisco_fwsm.log.10 does not exist
renaming /var/log/cisco_fwsm.log.9 to /var/log/cisco_fwsm.log.10 (rotatecount 14, logstart 1, i 9),
old log /var/log/cisco_fwsm.log.9 does not exist
renaming /var/log/cisco_fwsm.log.8 to /var/log/cisco_fwsm.log.9 (rotatecount 14, logstart 1, i 8),
old log /var/log/cisco_fwsm.log.8 does not exist
renaming /var/log/cisco_fwsm.log.7 to /var/log/cisco_fwsm.log.8 (rotatecount 14, logstart 1, i 7),
old log /var/log/cisco_fwsm.log.7 does not exist
renaming /var/log/cisco_fwsm.log.6 to /var/log/cisco_fwsm.log.7 (rotatecount 14, logstart 1, i 6),
old log /var/log/cisco_fwsm.log.6 does not exist
renaming /var/log/cisco_fwsm.log.5 to /var/log/cisco_fwsm.log.6 (rotatecount 14, logstart 1, i 5),
old log /var/log/cisco_fwsm.log.5 does not exist
renaming /var/log/cisco_fwsm.log.4 to /var/log/cisco_fwsm.log.5 (rotatecount 14, logstart 1, i 4),
old log /var/log/cisco_fwsm.log.4 does not exist
renaming /var/log/cisco_fwsm.log.3 to /var/log/cisco_fwsm.log.4 (rotatecount 14, logstart 1, i 3),
old log /var/log/cisco_fwsm.log.3 does not exist
renaming /var/log/cisco_fwsm.log.2 to /var/log/cisco_fwsm.log.3 (rotatecount 14, logstart 1, i 2),
old log /var/log/cisco_fwsm.log.2 does not exist
renaming /var/log/cisco_fwsm.log.1 to /var/log/cisco_fwsm.log.2 (rotatecount 14, logstart 1, i 1),
renaming /var/log/cisco_fwsm.log.0 to /var/log/cisco_fwsm.log.1 (rotatecount 14, logstart 1, i 0),
old log /var/log/cisco_fwsm.log.0 does not exist
log /var/log/cisco_fwsm.log.15 doesn't exist -- won't try to dispose of it
renaming /var/log/cisco_fwsm.log to /var/log/cisco_fwsm.log.1
running postrotate script

Then the cisco_fwsm.log file disappears:
[root@scylla log]# ll cisco_fwsm*
-rw-r--r-- 1 root root 27361305 Oct 2 19:23 cisco_fwsm.log.1
-rw-r--r-- 1 root root 187102161 Oct 2 02:01 cisco_fwsm.log.10.gz
-rw-r--r-- 1 root root 183803996 Oct 2 01:01 cisco_fwsm.log.11.gz
-rw-r--r-- 1 root root 178479419 Oct 2 00:01 cisco_fwsm.log.12.gz
-rw-r--r-- 1 root root 177124800 Oct 1 23:01 cisco_fwsm.log.13.gz
-rw-r--r-- 1 root root 165375931 Oct 1 22:01 cisco_fwsm.log.14.gz
-rw-r--r-- 1 root root 246071296 Oct 2 19:24 cisco_fwsm.log.17.gz
-rw-r--r-- 1 root root 137558120 Oct 2 11:01 cisco_fwsm.log.1.gz
-rw-r--r-- 1 root root 2021969431 Oct 2 19:14 cisco_fwsm.log.2
-rw-r--r-- 1 root root 144976378 Oct 2 10:01 cisco_fwsm.log.2.gz
-rw-r--r-- 1 root root 148809317 Oct 2 09:01 cisco_fwsm.log.3.gz
-rw-r--r-- 1 root root 150726072 Oct 2 08:01 cisco_fwsm.log.4.gz
-rw-r--r-- 1 root root 111259271 Oct 2 07:01 cisco_fwsm.log.5.gz
-rw-r--r-- 1 root root 42730667 Oct 2 06:01 cisco_fwsm.log.6.gz
-rw-r--r-- 1 root root 51935973 Oct 2 05:01 cisco_fwsm.log.7.gz
-rw-r--r-- 1 root root 178502282 Oct 2 04:01 cisco_fwsm.log.8.gz
-rw-r--r-- 1 root root 184110822 Oct 2 03:01 cisco_fwsm.log.9.gz


I have to restart msyslogd to have cisco_fwsm.log file again.

Thanks!
Zhenning

 

[zhenning]

It seems the current log files are rotate and compressed every hour. I do not how is that configured.

Thanks!
Zhenning

 

[ZaSter]

The reason the syslogd daemon is stopping is because your script kills it:

        postrotate
          /bin/kill -HUP `cat /var/run/[b][COLOR=red yellow]syslogd[/color][/b].pid 2> /dev/null` 2> /dev/null || true
        endscript

--
ZaSter

 

[zhenning]

But the server is using msyslogd, not syslogd. I just wondering how it is configured to rotate and compress every hour. It is not working like that any more.

Thanks!

 

[ZaSter]

Zhenning,

I am not sure, but I believe that the msyslogd uses the same PID file as does "syslogd", specifically, "/var/log/syslogd.pid". In any case, there is no reason for your log rotation script to kill HUP anything. I believe your script is actually making msyslogd die. Did you not notice that nothing was printed after "running postrotate script" was printed in the verbose run? That is because it ran the kill -HUP on the daemon that was running it. Log rotations do not require a postrotate script unless there is a specific need.

At least humor me and try removing the postrotate/endscript part of the script and see what happens.

The every hour compression can not happen if your script is killing the very daemon that is needed to produce the logs.

Any of this make any sense to you?

--
ZaSter

 

[Annihilannic]

kill -HUP does not kill syslogd, it only tells it to re-read its configuration, and close and reopen all log files.

Log rotation will only occur hourly if you are running logrotate from cron at least once an hour. Usually it only runs once a day... have you checked that?

Is this log definitely updated by msyslogd? If so, you'll need to find out whether msyslogd supports kill -HUP in the same way - you may be able to just use pkill -HUP msyslogd as your postrotate script.

If it does not support that, try using the copytruncate option.

Annihilannic.

 

[zhenning]

Yes logrotate is under /etc/cron.hourly folder. So it should run hourly.

Here is the logrotate and /etc/logrotate.conf files:

[root@scylla cron.hourly]# cat logrotate
#!/bin/sh

/usr/sbin/logrotate /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
/usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0

[root@scylla cron.hourly]# cat /etc/logrotate.conf
# see "man logrotate" for details
# old: rotate log files weekly
# old: weekly
#rotate log files monthly
monthly

# old: keep 4 weeks worth of backlogs
# old: rotate 4
# keep 12 months worth of backlogs
rotate 12

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
compress

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp -- we'll rotate them here
/var/log/wtmp {
monthly
create 0664 root utmp
rotate 1
}

# system-specific logs may be also be configured here.

I am sure it is using msyslog and the configuration is always working. Today I just modified the rotate in the following file:

[root@scylla cron.hourly]# cat /etc/logrotate.d/syslog_cisco_fwsm
/var/log/cisco_fwsm.log {
size=200M
rotate 14
postrotate
/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
endscript
}

Then run logrotate -f syslog_cisco_fwsm. Then it stopped working. Not sure how I can break it.

Zhenning

1. modify the file

 

[zhenning]

I just checked /var/mail/root file and I found the error:

/etc/cron.hourly/logrotate:

error: syslog_cisco_fwsm.old:1 duplicate log entry for /var/log/cisco_fwsm.log

I think the problem is when I modified the /etc/logrotate.d/syslog_cisco_fwsm file, I copied this file to syslog_cisco_fwsm.old and that is the problem. I just deleted the syslog_cisco_fwsm.old file and see if it will rotate for next hour.

Thanks!

 

[zhenning]

That is the problem. Thank you very much for all the helps!

Zhenning