python logging: how to ensure logfile directory is created?

Question

I would like to use python's logging framework in my application, and I'd like to allow the end user of my application to specify the log file. (Via the Python logging framework's configuration mechanisms which in my case is a section of a YAML file that the end user can edit to specify how logging behaves.)

Is there a way to get the logging framework to ensure that a directory exists by creating it? Because the exact path to the logging filename is embedded in the configuration information specified by the end user, it is nontrivial for me as the application writer to parse this information to determine which directory should be created.

If the end user specifies "foo/bar/baz.log", I would like to make sure that the foo/bar directory is created.

Note: This is the Python equivalent of this SO question about Java logging.

Gutturalize

Not as part of the logging framework itself. I'd suggest to just use a helper function to emulate mkdir -p and create the directory yourself. – Nuke

I know how to do that part, but I'm not in control of the log file name, so as the application writer, I can't create the directory myself. I was wondering if there was some kind of hook inside the logging framework to do this. – Gutturalize

Huh? You said the end user will specify the log file name - so that name would be passed to your application by means of a command line argument, config file or such, no? – Nuke

No, the name is passed to my application inside a config file, which I pass directly into the Python logging framework. Parsing this information is nontrivial, and I would have to recreate a lot of what the Python logging framework is doing. – Gutturalize

Oh, I see. Then unfortunately I don't see how that could be done, short of monkey patching that behavior onto the logging module. – Nuke

@LukasGraf: You don't need to monkeypatch, just subclass. Which I was writing up, but unutbu beat me to it (and with complete code and a shorter explanation to boot). – Barnstorm

Answer 1

Subclass FileHandler (or whatever handler you are using) to call mkdir_p during initialization:

import logging
import os
import errno

def mkdir_p(path):
    """https://mcmap.net/q/36311/-mkdir-p-functionality-in-python-duplicate (tzot)"""
    try:
        os.makedirs(path, exist_ok=True)  # Python>3.2
    except TypeError:
        try:
            os.makedirs(path)
        except OSError as exc: # Python >2.5
            if exc.errno == errno.EEXIST and os.path.isdir(path):
                pass
            else: raise

class MakeFileHandler(logging.FileHandler):
    def __init__(self, filename, mode='a', encoding=None, delay=0):            
        mkdir_p(os.path.dirname(filename))
        logging.FileHandler.__init__(self, filename, mode, encoding, delay)

Superhighway

Nice answer, but I think it would be a bit simpler to just use dirname = os.path.dirname(filename) instead of using split to get both dirname and basename and ignoring one of them basename. – Barnstorm

Oh, also, if you're using Python 3.2+, you don't need the fancy mkdir_p; just use os.makedirs with exist_ok=True. – Barnstorm

Looking at the links from the OP, you've suggested almost exactly the same thing as the top answer to the equivalent Java question (extend RollingFileAdapter as MyRollingFileAdapter and override its setFile method). Not too surprising, I guess, given that Python logging and java.util.logging are both based on log4j… – Barnstorm

There should be an optional argument called mode. The documentation is not clear about the mode to use to create the file... – Turdine