Python logging: create log if not exists or open and continue logging if it does

Question

I'm writing some code that uses the python logging system. The idea is that if the LOG doesn't already exist create the log but if it does then get the log and resume logging to that file. Here is my code:

import logging
import os

log_filename='Transactions.log')
if os.path.isfile(log_filename)!=True:
    LOG = logging.getLogger('log_filename')
    LOG.setLevel(logging.DEBUG)
    # create file handler which logs even debug messages
    fh = logging.FileHandler('log_filename')
    fh.setLevel(logging.DEBUG)
    # create console handler with a higher log level
    ch = logging.StreamHandler()
    ch.setLevel(logging.DEBUG)
    # create formatter and add it to the handlers
    formatter = logging.Formatter('-->%(asctime)s - %(name)s:%(levelname)s - %(message)s')
    fh.setFormatter(formatter)
    ch.setFormatter(formatter)
    # add the handlers to the logger
    LOG.addHandler(fh)
    LOG.addHandler(ch)
else:
    LOG=logging.getLogger()

I suspect the problem is with my else block but I don't know how to fix. Could anybody shed some light on this situation.

Ministrant

set up your logger once and then import LOG from any place you want, no need to check if the log file exists – Rudie

Answer 1

The logging module's FileHandler takes care of that for you. No need for complexity.

The handler takes an optional mode parameter, to specify whether it starts writing or appending data to it.

From the docs:

class logging.FileHandler(filename, mode='a', encoding=None, delay=False)

The specified file is opened and used as the stream for logging. If mode is not specified, 'a' is used.

Tamaru

If the directory does not exist, you will still get an exception. Logging is happy to create a new file for you, but not a whole directory structure. – Saree

Which mode should @Ministrant specify in order to create the file if not present? – Charinile

@Charinile if file is not present, either of a or w would be fine. a would append to the same file in next run, whereas w mode will delete the old file and create an entirely new one. Also read what @Saree commented above. – Tamaru

On Python 2.7, it doesn't seem to work. FileHandler can't find the path and it returns error unless you explicitly make a path using os.makedirs() – Hodometer

Answer 2

For anyone who was trying to create a new directory structure like logs/mylogfile.log, as @mightypile mentioned, FileHandler will not create a new directory structure for you. I used os.makedirs to ensure the directory structure.

import os
import logging

log_filename = "logs/output.log"
os.makedirs(os.path.dirname(log_filename), exist_ok=True)
file_handler = logging.FileHandler(log_filename, mode="w", encoding=None, delay=False)

Yah

Thanks! I believe it's log_filename on the last line – Defeatist

Answer 3

When you run

LOG = logging.getLogger('log_filename')

for the first time a global variable is created. Hence, you could also add the following code to the script above:

global LOG
if LOG is not None:
    print("found logger !")
else:
    ("no global variable logger found")

Lawgiver

Answer 4

You might want to use absolute path.

logging.FileHandler(filename="/home/user/PycharmProjects/project/logs/log_file.log")

Assimilate

Answer 5

Based on @KhoPhi and @loop 's answers, when using a logging config dict you might want to use a class as a handler that you can use in the config. For that you can use the following:

from pathlib import Path
from logging.handlers import RotatingFileHandler as _RotatingFileHandler

class RotatingFileHandler(_RotatingFileHandler):
    def __init__(self, filename: str, *args, **kwargs):
        Path(filename).absolute().parent.mkdir(exist_ok=True, parents=True)
        super().__init__(filename, *args, **kwargs)

you can then use all the examples above with your own handler:

RotatingFileHandler("logs/my_file.log")

or use it in the config file as:

...
handlers:
  my_handler:
    class: your_module.your_submodule.RotatingFileHandler
    filename: logs/my_file.log
...

Thyroiditis