Logrotate files with date in the file name

Question

I am trying to configure logrotate in RHEL for tomcat6 logs. Currently, logrotate works fine for catalina.out log, it is rotated and compressed properly.

The problem is with the files with date in them like:

catalina.2012-01-20.log
catalina.2012-01-21.log
catalina.2012-01-22.log

These files are not being rotated. I understand that I have to configure these in /etc/logrotate.d/tomcat6 file where rotation for catalina.out is configured. But I am not able to configure it.

All I want is these older files to be compressed daily, except the current date log file.

Can anybody help me out on this, please!!

Thanks Noman A.

Danny

Same problem, with tomcat. log-rotate sees each catalina.[date].log as a different file. It dose not group them as catalina.log. It is not rotating them. Each file comes out as catalina.[data].log.1.gz. So I end up with 200 files with the log.1.gz at the end. – Heathendom

Answer 1

(First post ever so if it looks like a drunk spider has formatted it then sorry)

After using our friend Google, here and I can't remember where else I managed to achieve something using logrotate (rather than cron or some other equivalent).

I have a the following in /var/log/rsync/:

-rw-r--r-- 1 root root 1.1M Apr  9 08:13 2014-04-09 07:48:18.log
-rw-r--r-- 1 root root 1.4M Apr 11 15:20 2014-04-11 15:02:52.log
-rw-r--r-- 1 root root 1.6M Apr 11 15:42 2014-04-11 15:22:04.log
-rw-r--r-- 1 root root 1.8M Apr 12 08:01 2014-04-12 07:45:31.log
-rw-r--r-- 1 root root 2.0M Apr 13 08:10 2014-04-13 07:53:38.log
-rw-r--r-- 1 root root 2.2M Apr 14 08:19 2014-04-14 07:51:09.log
-rw-r--r-- 1 root root 2.5M Apr 15 08:05 2014-04-15 07:37:38.log
-rw-r--r-- 1 root root 2.7M Apr 16 08:11 2014-04-16 07:43:14.log

and the following logrotate file:

/var/log/rsync/*.log {
       daily
       rotate 7
       compress
       delaycompress
       notifempty
       missingok
}

which I thought was perfectly reasonable. But after it refused to work and on finding out that it would never work (courtesy of this post) I wondered if it could be fudged to make it work.

After much testing and tweaking I managed to fudge it the following way:

/var/log/rsync/dummy {
        daily
        rotate 0
        create
        ifempty
        lastaction
                /usr/bin/find /var/log/rsync/ -mtime +7 -delete
                /usr/bin/find /var/log/rsync/ -mtime +1 -exec gzip -q {} \;
        endscript
}

into a logrotate config file called /etc/logrotate.d/local-rsync. Then create the dummy log file:

touch /var/log/rsync/dummy

then force a logrotate with:

logrotate -fv /etc/logrotate.d/local-rsync

which gives:

-rw-r--r-- 1 root root  71K Apr  9 08:13 2014-04-09 07:48:18.log.gz
-rw-r--r-- 1 root root  88K Apr 11 15:20 2014-04-11 15:02:52.log.gz
-rw-r--r-- 1 root root  82K Apr 11 15:42 2014-04-11 15:22:04.log.gz
-rw-r--r-- 1 root root  84K Apr 12 08:01 2014-04-12 07:45:31.log.gz
-rw-r--r-- 1 root root  87K Apr 13 08:10 2014-04-13 07:53:38.log.gz
-rw-r--r-- 1 root root  92K Apr 14 08:19 2014-04-14 07:51:09.log.gz
-rw-r--r-- 1 root root 2.5M Apr 15 08:05 2014-04-15 07:37:38.log
-rw-r--r-- 1 root root 2.7M Apr 16 08:11 2014-04-16 07:43:14.log
-rw-r--r-- 1 root root    0 Apr 16 12:11 dummy

Now just wait for tomorrow morning...

I realise that cron would be tidier however I have another element in the logrotate config file and wanted to keep the two together.

Bonus with the dummy file is that it doesn't take up any space!

You may find that it does not appear to have rotated anything one day. It took me while to work out why but then it twigged. find -mtime +1 is whole days (i.e. 24*60 minutes) and if the daily logrotate kicked in less than 24 hours since the last time/time the logs were created then it sometimes appears not to have worked. If it bothers you then using 23 hours with find -mmin +1380 might be more appropriate.

Bookout

With the lastaction/endscript option you saved me! Thanks – Apatite

i voted the answer just for the very first line ... didnt even read further to verify solution is correct. – Kerley

How can this be applied for Windows ? – Emmerie

Answer 2

I spent a quite a while reading a lot of documentation. Logrotate does not seem to be able to group the different files with dates included in the name of the file. Logrotate can not do what we need it to do.

You have two options change the logging facility provided by java / tomcat to not include the date in the file name. http://tomcat.apache.org/tomcat-6.0-doc/logging.html

The second and quicker way is to use your own little script to do the work for you, using find. https://serverfault.com/questions/256218/logrotation-when-filenames-includes-date, https://serverfault.com/a/256231/71120

find /pathtologs/* -mtime +5 -exec rm {} \;

I went with the second option, because our developers have coded for dates in the files names. So it needs to stay that way. The -mtime +5 sets find to only look for files who are older then 5 days.

From find's documentation.

File's data was last modified n*24 hours ago. See the comments for -atime to understand how rounding affects the interpretation of file modification times.

Updated as per comment

find /pathtologs/* -mtime +5 -delete

If you specifically want to delete, this is a quick way to do it. If you need to some other command you can always replace the exec rm {} \; with something else.

Heathendom

instead of doing -exec rm {} \;, you can also simply do -delete in many versions of find. I think it's clearer. – Signorelli

@LászlóvandenHoek Just be sure you're doing the same thing. -delete is not rm. -delete implies -depth. Unlike rm, it will delete directories if they are empty (or being emptied by the command). I agree though, -delete is nice. Didn't know it existed. Thanks. – Secundines

Answer 3

Something like this in /etc/cron.d/rotate_tomcat_logs:

# delete every log file over 100 days old, and compress every log file over 1 day old.
00 1 * * * root ( find /opt/tomcat/logs -name \*log\* -name \*.gz -mtime +100 -exec rm -f {} \; >/dev/null 2>&1 )
05 1 * * * root ( find /opt/tomcat/logs -name \*log\* ! -name \*.gz -mtime +1 -exec gzip {} \; >/dev/null 2>&1 )

Sweetie

Answer 4

/path/to/logs/*.log { missingok compress rotate 7 }

this type of thing doesn't work normally because as others point out tomcat has its own log rotation. You can either use a simple cron to delete old files or turn off rotation on the access log valve. By turning off log rotation (and possible changing the filename patter), the above logrotate and other similar configs will work fine.

The bottom line is you should use logrotate or the built in log rotation in tomcat but not both at the same time.

Cake

Answer 5

To include a date in the rotated file, you can probably use 'dateext' option.

$ cat logrotate.conf 
/var/nginx/logs/access.log {
    size 10k
    copytruncate
    dateext
    rotate 10
    compress
}

The rotated file should get created similar to below

 root@nitpc:~# ls -lrt /var/nginx/logs/access.*
-rw-r--r-- 1 nginx root 5422 May 31 08:26 access.log
-rw-r--r-- 1 nginx root  466 May 31 08:26 access.log-20180531.gz

The only downside is you won't be able to run it more than once per day as the file would have a definite name for that date.

The above example is from my Nginx docker container running in k8s on GC. The logrotate version is 3.11.0.

Hope that helps!

Update: From man pages https://linux.die.net/man/8/logrotate

dateformat format string

Specify the extension for dateext using the notation similar to strftime(3) function. Only %Y %m %d and %s specifiers are allowed. The default value is -%Y%m%d. Note that also the character separating log name from the extension is part of the dateformat string. The system clock must be set past Sep 9th 2001 for %s to work correctly. Note that the datestamps generated by this format must be lexically sortable (i.e., first the year, then the month then the day. e.g., 2001/12/01 is ok, but 01/12/2001 is not, since 01/11/2002 would sort lower while it is later). This is because when using the rotate option, logrotate sorts all rotated filenames to find out which logfiles are older and should be removed.

Picrite

This is not the point of the question. The generated log files already have date format. The idea is not to add a date, but to manage files that were already generated with a date. – Holst

You are actually right. Thanks for pointing out that my reply doesn't answer the question directly. Nonetheless something mentioned in it can be used hence not updating it. – Picrite

Answer 6

Also, you can add crons instead hardcode logrotate.

1 0 * * * /usr/bin/find /var/log/tomcat/ -mtime +30 -delete
2 0 * * * /usr/bin/find /var/log/tomcat/ -mtime +1 -exec gzip -q {} \;

In this way, you will delete the logs older than 30 days, and zip the logs older than 1.

Disenthrall

thanks! it was good for me. gzip -q is the best option) I only used -exec rm -rf {} \; instead of -delete in the first record. – Tewfik

Answer 7

Probably you can remove the date from the log file names as described in How to remove the date pattern from tomcat logs to be able to use logrotate rules.

This worked for me at least for localhost access log.

Thaler

Answer 8

Well, I was not fully satisfied with any of the answers, even though the ones stating that the logrotate doesn't support this (i. e. just to remove files rotated by some other application) scenario are surely correct (raise a feature request on that tool, maybe?).

So, I would like to share an alternative approach I've come to. Unlike the "find /path/to/logs -mtime +7 -delete" solution, this one won't remove all the old logs after a specified period of time. So here it comes, an example one-liner bash command which leaves just N last logs on the disk (whenever it is run):

for f in `ls -1r | grep -E "^catalina\.[0-9]{4}-[0-9]{2}-[0-9]{2}\.log$" | tail -n +$((N+1))`; do rm $f; done

Finally, to cover the topic completely, the last alternative solution is not to rotate the log files (e. g. use rotatable=false in case of Tomcat - see its docs) and use the logrotate as usually, but don't forget to use it with the 'copytruncate' option.

Suggestions are welcome...

Mining

Answer 9

In your log rotate file, use rotate #, where # is the number of logs you want to keep before removing them.

rotate count

Log files are rotated times before being removed or mailed to the address specified in a mail directive. If count is 0, old versions are removed rather then rotated.

Encroach

Thanks for the reply Paul. I am passing file name as /var/log/tomcat6/catalina*.log. since the logfile name is not conistent (it has dates in it). And these files are rolled over daily by default by juli (tomcat's default). I just want to compress them using logrotate, leaving the current date log untouched and uncompressed. Hope my query is clear. Thanks for the reply again! – Danny

The current day's log should always be left untouched. That's what logrotate does. By setting rotate #, it will remove the old ones after # many have been rotated out. If you want to compress them, as well, use the compress command in your rotate script. linuxcommand.org/man_pages/logrotate8.html – Encroach

Hi Paul, sorry for the late comment, but I was testing this. So my config is /var/log/tomcat6/catalina*.log { daily rotate 30 compress missingok nocreate nodateext } So I assume the log should be rotated, but it is not being rotated. When I manually ran logrotate in verbose mode, got the following output: "considering log /var/log/tomcat6/catalina.2012-01-23.log log does not need rotating considering log /var/log/tomcat6/catalina.2012-01-24.log log does not need rotating Am I doing anything wrong? – Danny

Answer 10

To enable daily log rotation in tomcat [linux] with postfix of date to catalina file, do the below changes.

Download cronolog package and install in linux os

wget http://pkgs.fedoraproject.org/repo/pkgs/cronolog/cronolog-1.6.2.tar.gz/md5/a44564fd5a5b061a5691b9a837d04979/cronolog-1.6.2.tar.gz

tar -zxvf cronolog-1.6.2.tar.gz
./configure
make
make install 

open apache-tomcat-7.0.65/bin/catalina.sh file using vi command and change the lines as given below example : /opt/apache-tomcat-7.0.65/bin/catalina.sh

shift
touch "$CATALINA_OUT"
# comment above line as below 
#touch "$CATALINA_OUT"
if [ "$1" = "-security" ] ; then
if [ $have_tty -eq 1 ]; then
  echo "Using Security Manager"
fi
shift
eval "\"$_RUNJAVA\"" "\"$LOGGING_CONFIG\"" $LOGGING_MANAGER $JAVA_OPTS $CATALINA_OPTS \
  -Djava.endorsed.dirs="\"$JAVA_ENDORSED_DIRS\"" -classpath "\"$CLASSPATH\"" \
  -Djava.security.manager \
  -Djava.security.policy=="\"$CATALINA_BASE/conf/catalina.policy\"" \
  -Dcatalina.base="\"$CATALINA_BASE\"" \
  -Dcatalina.home="\"$CATALINA_HOME\"" \
  -Djava.io.tmpdir="\"$CATALINA_TMPDIR\"" \
  org.apache.catalina.startup.Bootstrap "$@" start \
  #>> "$CATALINA_OUT" 2>&1 "&"
  # comment above line and add below given line
2>&1 |/usr/local/sbin/cronolog "$CATALINA_BASE/logs/catalina-%Y-%m-%d.out" &
else
eval "\"$_RUNJAVA\"" "\"$LOGGING_CONFIG\"" $LOGGING_MANAGER $JAVA_OPTS $CATALINA_OPTS \
  -Djava.endorsed.dirs="\"$JAVA_ENDORSED_DIRS\"" -classpath "\"$CLASSPATH\"" \
  -Dcatalina.base="\"$CATALINA_BASE\"" \
  -Dcatalina.home="\"$CATALINA_HOME\"" \
  -Djava.io.tmpdir="\"$CATALINA_TMPDIR\"" \
  org.apache.catalina.startup.Bootstrap "$@" start \
  #>> "$CATALINA_OUT" 2>&1 "&"         
  # comment above line and add below given line
2>&1 |/usr/local/sbin/cronolog "$CATALINA_BASE/logs/catalina-%Y-%m-%d.out" &
fi

After the above changes save the file and restart the tomcat to apply changes.

Mccurry

This answer has complicated changes and does not explain what and why is done. – Primateship

Answer 11

The current answers may work but I find they make things too complicated instead of using logrotate in a way where it already handles things very well. You don't need to specify commands manually if logrotate already handles this. You don't need complicated configuration if you just address the initial problem of catalina doing the logrotation and let logrotate handle it.

---

It looks like the files are already being rotated as illustrated by the date in the logfile names in the question:

catalina.2012-01-20.log
catalina.2012-01-21.log
catalina.2012-01-22.log

So, which component already does this, is it configured in /etc/logrotate.d/tomcat*? What is the content of this file on your system currently? If I remember correctly, catalina may already doing this itself: Creating a new logfile every day and moving older files to their new location. If you want logrotate to handle this instead, you may want to change the default behaviour of tomcat / catalina, as described in this answer first.

After that, catalina should always write to catalina.log.

Then you can put the rotation / compression / deletion entirely into the responsibility of logrotate.

If you want to compress older files, you can add compress to the logrotate configuration:

For example, you have:

/somepath/catalina.log {
  # truncate file in place
  copytruncate
  # rotate daily
  daily
  # keep max 7, will delete after that
  rotate 7
  # when rotating create file with date, e.g. catalina.log-20200304
  dateext
  # compress
  compress
  # manpage: "Postpone  compression  of  the previous log file
  #   to the next rotation cycle.  
  #   This only has effect when used in combination with compress."
  delaycompress 
  # change this to correct owner and permissions of logfile
  create 750 tomcat6 tomcat6
  # do not generate an error if logfile is missing
  missingok
  # do not rotate if the file is empty
  notifempty 

}

What you should get as a result:

catalina.log
catalina.log-20200304
catalina.log-20200303.gz
...

If things are not working:

• execute logrotate manually with -v
• It should also show the status file where the last rotate is logged, e.g. /var/lib/logrotate/status
• Usually it will rotate once a day (or whatever you specified). You can force this with -f. Or you can manipulate the status file (do not do this in production unless you know what you are doing)

A common error is that people run logrotate manually and then get puzzled if nothing is changed. Logrotate will only perform the operations it has been configured to perform. If you specify a daily rotate, it will not rotate again even if you run it as often as you want, unless you use -f. It will use the status file to keep track. You can use -v and logrotate will report what it is doing and why.

Resources:

• logrotate man page

Ugaritic