python logging - how do I truncate the pathname to just the last few characters or just the filename?

Question

I'm using python logging and I have a formatter that looks like the following:

formatter = logging.Formatter(
    '%(asctime)s - %(pathname)86s - %(lineno)4s - %(message)s', '%d %H:%M'
    )

As you can see, I like the information in my log files to line up neatly in columns. The reason I have 86 spaces reserved for the pathname is because the full paths to some of the files used in my program are that long. However, all I really need is the actual file name, not the full path. How can I get the logging module to give me just the file name? Better yet, since I have some long filenames, I'd like the first 3 characters of the filename, followed by '~', then the last 16 characters. So

/Users/Jon/important_dir/dev/my_project/latest/testing-tools/test_read_only_scenarios_happily.py

should become

tes~arios_happily.py

Udela

I saw your edit suggestion; glad you worked it out. record.args is either a dict or a tuple, but your formatter suggested that you were using a dict exclusively, which is why I created my solution based on that. It's easy enough to adapt to a tuple too, as you discovered. :-) – Elaterin

Answer 1

You'll have to implement your own Formatter subclass that truncates the path for you; the formatting string cannot do this.

Since you want to use just the filename you'd want to use the filename parameter as the base, which we can then truncate as needed.

For the size-limited version, I'd actually use a new record attribute for, leaving the existing filename to be the full name. Lets call it filename20.

import logging

class FilenameTruncatingFormatter(logging.Formatter):
    def format(self, record):
        # truncate the filename if it is longer than 20 characters
        filename = record.filename
        if len(filename) > 20:
            filename = '{}~{}'.format(filename[:3], filename[-16:])
        record.filename20 = filename
        return super(PathTruncatingFormatter, self).format(record)

Use this class instead of the normal logging.Formatter instance, and reference the filename20 field in the formatting string:

formatter = FilenameTruncatingFormatter(
    '%(asctime)s - %(filename20)20s - %(lineno)4s - %(message)s', '%d %H:%M'
)

Note that I also shortened the space needed for the field to 20 characters: %(...)20s.

If you have to specify the formatter in a ConfigParser-style configuration file, create a named formatter via a [formatter_NAME] section, and give that section a class key:

[formatter_truncated]
class=some.module.FilenameTruncatingFormatter
format=%(asctime)s - %(filename20)20s - %(lineno)4s - %(message)s
datefmt=%d %H:%M

and then you can use formatter=truncated in any handler sections you define.

For the dictionary-schema style configuration, the same applies: specify the formatter class via the class key in any formatter definition.

Elaterin

Is there a way to do this if i am using logging.ini file ? – Fornication

@SouravPrem: no, because you can't define a custom class in logging.ini. – Elaterin

@SouravPrem: I see that you can now use a class key in the .ini style configuration format, so I can now state that you can do this in a logging.ini file by specifying where to import the custom PathTruncatingFormatter class from. I've added this to the answer. – Elaterin

This doesn't work for my. It can't find the module I put for class= and if I do fully qualified path for my class I get a circular import error. – Isogamete

Answer 2

You can simply use filename instead of pathname. Refer https://docs.python.org/3.1/library/logging.html#formatter-objects for more details

Eonism

Answer 3

Like this:

import os

def shorten_filename(filename):
    f = os.path.split(filename)[1]
    return "%s~%s" % (f[:3], f[-16:]) if len(f) > 19 else f

Tensiometer

Minor nit, I'd use len(f) > 20, since the shortened file name will have 20 characters anyway (19 taken from the file name plus the tilde). – Appellate

Thanks. However, this answer doesn't address the problem in the context of the logging module. – Udela

Answer 4

Python 3: Same but different, but thought I would share, I simply wanted the last two things in the path, The directory and the file name.

class MyFormatter(logging.Formatter):
    def format(self, record):
        if 'pathname' in record.__dict__.keys():
            sections = record.pathname.split("/")
            sec_len = len(sections)
            record.pathname = "{}/{}".format(sections[sec_len - 2], sections[sec_len - 1])
        return super(MyFormatter, self).format(record)

Burger

Answer 5

For python 3.8 and above :

Same than response from @Glen Bizeau but in a more compact and tuneable way, allowing to set the number of maximum parent folders displayed :

class FileFormatter(logging.Formatter):
    MAX_FOLDERS = 4

    def format(self, record):
        if 'pathname' in record.__dict__.keys():
            if (sec_len := len(sections := record.pathname.split(os.path.sep))) > self.MAX_FOLDERS :
                record.pathname = "(...)" + os.path.sep.join([sections[sec_len - i] for i in reversed(range(1,self.MAX_FOLDERS+1))])
        return super().format(record)

Yields: (...)__packages__\Inflow\ios\load.py
instead of: C:\Users\MyName\Documents\Python\__packages__\Inflow\ios\load.py

Allayne

Answer 6

The following is the Python 3 version of the same which @Martijn had written. As @Nick pointed out, the record parameter passed is a LogRecord object in Python 3.

import logging
import os

class PathTruncatingFormatter(logging.Formatter):
    def format(self, record):
        if 'pathname' in record.__dict__.keys():
            # truncate the pathname
            filename = os.path.basename(record.pathname)
            if len(filename) > 20:
                filename = '{}~{}'.format(filename[:3], filename[-16:])
            record.pathname = filename
        return super(PathTruncatingFormatter, self).format(record)

Sannyasi