How to use Python's RotatingFileHandler

Question

I'm trying to do a test run of the logging module's RotatingFileHandler as follows:

import logging
from logging.handlers import RotatingFileHandler

# logging.basicConfig(filename="example.log", level=logging.DEBUG)

logger = logging.getLogger('my_logger')
handler = RotatingFileHandler("my_log.log", maxBytes=2000, backupCount=10)
logger.addHandler(handler)

for _ in range(10000):
    logger.debug("Hello, world!")

However, with logging.basicConfig line commented out, the resulting my_log.log file contains no data:

If I comment in the line with logging.basicConfig(filename="example.log", level=logging.DEBUG), I get the expected my_log.log files with numbered suffixes. However, there is also the example.log which is a (relatively) large file:

How can I set up the logging so that it only generates the my_log.log files, and not the large example.log file?

Hermann

Answer 1

Python provides 5 logging levels out of the box (in increasing order of severity): DEBUG, INFO, WARNING, ERROR and CRITICAL. The default one is WARNING. The docs says, that

Logging messages which are less severe than lvl will be ignored.

So if you use .debug with the default settings, you won't see anything in your logs.

The easiest fix would be to use logger.warning function rather than logger.debug:

import logging
from logging.handlers import RotatingFileHandler

logger = logging.getLogger('my_logger')
handler = RotatingFileHandler('my_log.log', maxBytes=2000, backupCount=10)
logger.addHandler(handler)

for _ in range(10000):
    logger.warning('Hello, world!')

And if you want to change logger level you can use .setLevel method:

import logging
from logging.handlers import RotatingFileHandler

logger = logging.getLogger('my_logger')
logger.setLevel(logging.DEBUG)
handler = RotatingFileHandler('my_log.log', maxBytes=2000, backupCount=10)
logger.addHandler(handler)

for _ in range(10000):
    logger.debug('Hello, world!')

Liquidize

At first it wasn't clear to me how this answers the question, but key point is that log messages which are not above the level set for the logger (in the debug(), info(), warning(), error(), exception(), critical() hierarchy) do not get logged (in this case, written to the output file). – Hermann

Answer 2

Going off of Kurt Peek's answer you can also put the rotating file handler in the logging.basicConfig directly

import logging
from logging.handlers import RotatingFileHandler
logging.basicConfig(
        handlers=[RotatingFileHandler('./my_log.log', maxBytes=100000, backupCount=10)],
        level=logging.DEBUG,
        format="[%(asctime)s] %(levelname)s [%(name)s.%(funcName)s:%(lineno)d] %(message)s",
        datefmt='%Y-%m-%dT%H:%M:%S')

Petiole

what's the difference b/w rotating file handler and file handler. – Mahau

FileHandler sends logging output to a disk file. RotatingFileHandler supports sending logging to a rotation of disk log files. – Petiole

Answer 3

All previous answers are correct, here another way of doing the same thing except we use logging config file instead.

logging_config.ini

Here is the config file :

[loggers]
keys=root

[handlers]
keys=logfile

[formatters]
keys=logfileformatter

[logger_root]
level=DEBUG
handlers=logfile

[formatter_logfileformatter]
format=%(asctime)s %(name)-12s: %(levelname)s %(message)s

[handler_logfile]
class=handlers.RotatingFileHandler
level=DEBUG
args=('testing.log','a',10,100)
formatter=logfileformatter

myScrypt.py

here is simple logging script that uses the above config file

import logging
from logging.config import fileConfig

fileConfig('logging_config.ini')
logger = logging.getLogger()
logger.debug('the best scripting language is python in the world')

RESULT

here is the result, notice maxBytes is set to 10 but in real life, that's clearly too small. (args=('testing.log','a',10,100)

Sweptback

for anyone else 'a' means append. docs.python.org/3/library/functions.html#filemodes – Gauguin

also the 10 is the size in bytes and the 100 is how many backups args=('testing.log','a',10,100) – Gauguin

@grepit, will you be able to provide any link to docs with all such availabe keys or the things in square braces, and how to write such .ini files! – Mahau

Answer 4

I found that to obtain the desired behavior one has to use the same name in the basicConfig and RotatingFileHandler initializations:

import logging
from logging.handlers import RotatingFileHandler

logging.basicConfig(filename="my_log.log", level=logging.DEBUG)

logger = logging.getLogger('my_logger')
handler = RotatingFileHandler("my_log.log", maxBytes=2000, backupCount=10)
logger.addHandler(handler)

for _ in range(10000):
    logger.debug("Hello, world!")

Here, I have chose the same name my_log.log. This results in only the 'size-limited' logs being created:

Hermann

In my 'real' application, I would actually like to have two loggers: one which prints to the console, and one which is saved to a file. I've read that in such cases, one should not use basicConfig, which is meant for if there is a single logger. So I would still be interested why this code doesn't work without the basicConfig line. – Hermann

Look at my answer, it has an explanation, why you code doesn't work without the basicConfig line. – Liquidize