How to perform log rotation with Gunicorn?

Question

I searched through the net but didn't get the concrete answer or example of "how to use log rotation with Gunicorn?".
It would be great if someone provide an example.

Rochellrochella

Answer 1

If you do not want to bother with logrotare you can use a full python/gunicorn solution using python logging facilities.

Create a file named log.conf with the content below. This will rotate the error log file and access log file daily and will keep the logs for 90 days. Log level is set to INFO.

Than, start gunicorn adding a command line parameter --log-config log.conf. Remove the --access-logfile, --error-logfile and --log-level parameters, as the log.conf will take care of everything.

More info on how to configure the logger is available in the Python documentation.

Here below is the content of log.conf. For another example look at gunicorn source code.

HTH

[loggers]
keys=root, gunicorn.error, gunicorn.access

[handlers]
keys=console, error_file, access_file

[formatters]
keys=generic, access

[logger_root]
level=INFO
handlers=console

[logger_gunicorn.error]
level=INFO
handlers=error_file
propagate=1
qualname=gunicorn.error

[logger_gunicorn.access]
level=INFO
handlers=access_file
propagate=0
qualname=gunicorn.access

[handler_console]
class=StreamHandler
formatter=generic
args=(sys.stdout, )

[handler_error_file]
class=logging.handlers.TimedRotatingFileHandler
formatter=generic
args=('/var/log/gunicorn/gunicorn-error.log', 'midnight', 1, 90, 'utf-8')

[handler_access_file]
class=logging.handlers.TimedRotatingFileHandler
formatter=access
args=('/var/log/gunicorn/gunicorn-access.log', 'midnight', 1, 90, 'utf-8')

[formatter_generic]
format=%(asctime)s [%(process)d] [%(levelname)s] %(message)s
datefmt=%Y-%m-%d %H:%M:%S
class=logging.Formatter

[formatter_access]
format=%(message)s
class=logging.Formatter

Legaspi

Answer 2

Gunicorn's documentation says you can setup log rotation with logrotate (a linux command):

Logs can be automatically rotated and compressed using logrotate.

Doc link: http://docs.gunicorn.org/en/latest/install.html?highlight=logrotate#debian-gnu-linux

So I guess Gunicorn provides itself no way to rotate log.

Here is an example of my configuration file, placed in /etc/logrotate.d/my_app:

/path/to/my/logs/gunicorn-access.log /path/to/my/logs/gunicorn-error.log {
    monthly
    dateext
    dateformat -%Y-%m
    dateyesterday
    rotate 10000
}

Rotate monthly, add -YEAR-MONTH to rotated files, keep 10000 rotated files (see man logrotate).

The paths at first line are declared in my gunicorn_start script, something like:

/my/virtualenv/bin/gunicorn OPTIONS \
    --access-logfile /path/to/my/logs/gunicorn-access.log \
    --error-logfile /path/to/my/logs/gunicorn-error.log

Kasper

Where do we specify gunicorn to use the settings from my_app? – Biomass

You don't. The log rotation is done by logrotate, not gunicorn. – Kasper

Answer 3

doc link : https://docs.gunicorn.org/en/latest/deploy.html#logging

Logging
Logging can be configured by using various flags detailed in the configuration documentation or by creating a logging configuration file. Send the USR1 signal to rotate logs if you are using the logrotate utility:
kill -USR1 $(cat /var/run/gunicorn.pid)

so you can write the configuration file like this:

/yourpath/log/gunicorn.* {
daily
rotate 30
compress
dateext
dateformat .%Y-%m-%d
notifempty
sharedscripts
postrotate
    kill -USR1 $(cat /yourpath/run/gunicorn.pid)
endscript
}

rotate everyday

Insecurity

This worked. I am running gunicorn via supervisor, so I had to put a supervisordctl restart <program name> in the postrotate script. Feels like a hack, but works for now. – Cornea

Answer 4

Adding a change to Charlie's approach, logging.handlers.TimedRotatingFileHandler doesn't work in recent python(3.8), so create a file,

simple_logger.py

from logging.handlers import TimedRotatingFileHandler

and update class line in log.conf,

class=simple_logger.TimedRotatingFileHandler

It worked!

NOTE: Rotation works but data pushed to logs is not in sequential.

Amye