Custom Logger class and correct line number/function name in log

Question

I'd like to wrap Python logger in a custom class to embed some application-specific functionality and hide setup details from developers (setting file output, logging level, etc). To do this, I created a class with the following API:

__init__(log_level, filename)
debug(msg)
info(msg)
warning(msg)
error(msg)

Logger.debug/info/warning/etc calls usually write in the log the function and line number where the log call was made. However, using my custom class, the function and line numbers written to the log file are always the same (correspondent to the debug()/info()/warning()/error() functions inside the custom class). I want it to save the line of application code that logged the msg. Is that possible?

Bile

Perhaps you could, instead, customize Handler or Formatter to embed app-specific functionality, and offer a stand-alone factory function to hide the setup details? – Nonie

@Alan Evangelista, could you please mark my answer correct? Thanks. – Equidistance

Answer 1

Try stacklevel, which counts the number of calls from the original logging call to the logger's debug(), info(), etc., call. It's new in logging 3.8:

The third optional keyword argument is stacklevel, which defaults to 1. If greater than 1, the corresponding number of stack frames are skipped when computing the line number and function name set in the LogRecord created for the logging event. This can be used in logging helpers so that the function name, filename and line number recorded are not the information for the helper function/method, but rather its caller. The name of this parameter mirrors the equivalent one in the warnings module.

Equidistance

This is the correct answer @Alan Evangelista, it requires no patching, no reproduction of code, nothing that might break later, simply pass a stacklevel kwarg when you call the underlying logger info, trace, debug (etc) calls and it will just work. – Funeral

Yes, that's the right approach. Replacing the findCaller() function is a slippery business and is no longer needed. – Admeasurement

Answer 2

It is possible to generate a log wrapper if you are willing to re-implent a little bit of the standard logging module. The trick is to write your own findCaller() method that knows how to ignore your logging-wrapper source-file when interpreting back-traces.

in logwrapper.py:

import logging
import os
import sys

from logging import *


# This code is mainly copied from the python logging module, with minor modifications

# _srcfile is used when walking the stack to check when we've got the first
# caller stack frame.
#
if hasattr(sys, 'frozen'): #support for py2exe
    _srcfile = "logging%s__init__%s" % (os.sep, __file__[-4:])
elif __file__[-4:].lower() in ['.pyc', '.pyo']:
    _srcfile = __file__[:-4] + '.py'
else:
    _srcfile = __file__
_srcfile = os.path.normcase(_srcfile)


class LogWrapper(object):
    def __init__(self, logger):
        self.logger = logger

    def debug(self, msg, *args, **kwargs):
        """
        Log 'msg % args' with severity 'DEBUG'.

        To pass exception information, use the keyword argument exc_info with
        a true value, e.g.

        logger.debug("Houston, we have a %s", "thorny problem", exc_info=1)
        """
        if self.logger.isEnabledFor(DEBUG):
            self._log(DEBUG, msg, args, **kwargs)

    def info(self, msg, *args, **kwargs):
        """
        Log 'msg % args' with severity 'INFO'.

        To pass exception information, use the keyword argument exc_info with
        a true value, e.g.

        logger.info("Houston, we have a %s", "interesting problem", exc_info=1)
        """
        if self.logger.isEnabledFor(INFO):
            self._log(INFO, msg, args, **kwargs)


    # Add other convenience methods

    def log(self, level, msg, *args, **kwargs):
        """
        Log 'msg % args' with the integer severity 'level'.

        To pass exception information, use the keyword argument exc_info with
        a true value, e.g.

        logger.log(level, "We have a %s", "mysterious problem", exc_info=1)
        """
        if not isinstance(level, int):
            if logging.raiseExceptions:
                raise TypeError("level must be an integer")
            else:
                return
        if self.logger.isEnabledFor(level):
            self._log(level, msg, args, **kwargs)


    def _log(self, level, msg, args, exc_info=None, extra=None):
        """
        Low-level logging routine which creates a LogRecord and then calls
        all the handlers of this logger to handle the record.
        """
        # Add wrapping functionality here.
        if _srcfile:
            #IronPython doesn't track Python frames, so findCaller throws an
            #exception on some versions of IronPython. We trap it here so that
            #IronPython can use logging.
            try:
                fn, lno, func = self.findCaller()
            except ValueError:
                fn, lno, func = "(unknown file)", 0, "(unknown function)"
        else:
            fn, lno, func = "(unknown file)", 0, "(unknown function)"
        if exc_info:
            if not isinstance(exc_info, tuple):
                exc_info = sys.exc_info()
        record = self.logger.makeRecord(
            self.logger.name, level, fn, lno, msg, args, exc_info, func, extra)
        self.logger.handle(record)


    def findCaller(self):
        """
        Find the stack frame of the caller so that we can note the source
        file name, line number and function name.
        """
        f = logging.currentframe()
        #On some versions of IronPython, currentframe() returns None if
        #IronPython isn't run with -X:Frames.
        if f is not None:
            f = f.f_back
        rv = "(unknown file)", 0, "(unknown function)"
        while hasattr(f, "f_code"):
            co = f.f_code
            filename = os.path.normcase(co.co_filename)
            if filename == _srcfile:
                f = f.f_back
                continue
            rv = (co.co_filename, f.f_lineno, co.co_name)
            break
        return rv

And an example of using it:

import logging
from logwrapper import LogWrapper

logging.basicConfig(level=logging.DEBUG, format="%(asctime)s %(filename)s(%(lineno)d): "
                    "%(message)s")
logger = logging.getLogger()
lw = LogWrapper(logger)

lw.info('Wrapped well, this is interesting')

Sesquipedalian

Answer 3

Based on @Will Ware's answer. Another option is to overwrite findCaller method and use custom class as default logger:

class MyLogger(logging.Logger):
    """
        Needs to produce correct line numbers
    """
    def findCaller(self):
        n_frames_upper = 2
        f = logging.currentframe()
        for _ in range(2 + n_frames_upper):  # <-- correct frame
            if f is not None:
                f = f.f_back
        rv = "(unknown file)", 0, "(unknown function)"
        while hasattr(f, "f_code"):
            co = f.f_code
            filename = os.path.normcase(co.co_filename)
            if filename == logging._srcfile:
                f = f.f_back
                continue
            rv = (co.co_filename, f.f_lineno, co.co_name)
            break
        return rv

logging.setLoggerClass(MyLogger)
logger = logging.getLogger('MyLogger')  # init MyLogger
logging.setLoggerClass(logging.Logger) # reset logger class to default

Celisse

Answer 4

Yes: sys._getframe(NUM) where NUM says how how many functions outside the current one you are looking for. The returned frame object has attributes like f_lineno and f_code.co_filename.

http://docs.python.org/library/sys.html#sys._getframe

Interracial

These functions are very useful, but there seems to be no way to integrate these with Python logging package. Therefore, I need to choose one of the following options: (1) customize Python logging package for my needs, (2) stop using Python logging package and reimplement a part of it in my logger class, (3) keep a Python logger inside my class and always reference it from outside to log messages (eg self.mylogger.logger.debug() or (4) drop my logger class, use Python logger directly, create a factory for it and customize handler/formatter (as suggested in the comment by Janne Karila). – Bile

Answer 5

The best place to change lineno and filename (if you really want to do that) is in a filter that is attached to the logger. Here is a proof-of-concept. The last line will log a customized message with filename and line of the last line:

import logging
import inspect
from pathlib import Path

class up_stacked_logger:
    def __init__(self, logger, n):
        self.logger = logger

        calling_frame = inspect.stack()[n+1].frame
        trace = inspect.getframeinfo(calling_frame)  

        class UpStackFilter(logging.Filter):
            def filter(self, record):       
                record.lineno = trace.lineno
                record.pathname = trace.filename
                record.filename = Path(trace.filename).name
                return True

        self.f = UpStackFilter()

    def __enter__(self):
        self.logger.addFilter(self.f)
        return self.logger

    def __exit__(self, *args, **kwds):
        self.logger.removeFilter(self.f)


def my_cool_customized_log_function(logger, msg):  

    with up_stacked_logger(logger, n=1) as logger:
        logger.info(f'--> customize {msg}')


logging.basicConfig(level=logging.DEBUG, format="[%(name)s][%(levelname)s]  %(message)s (%(filename)s:%(lineno)d)")
logger = logging.getLogger('some.module.logger')  

my_cool_customized_log_function(logger, 'a message')

Frere

Answer 6

Here's yet another stab at rewriting findCaller. This lets you customize the extra stack frame depth on a per-function basis.

import os
import logging
from contextlib import contextmanager

logging.basicConfig(
    format='%(asctime)-15s  %(levelname)s  %(filename)s:%(lineno)d  %(message)s',
    level=logging.INFO
)


@contextmanager
def logdelta(n, level=logging.DEBUG):
    _frame_stuff = [0, logging.Logger.findCaller]

    def findCaller(_):
        f = logging.currentframe()
        for _ in range(2 + _frame_stuff[0]):
            if f is not None:
                f = f.f_back
        rv = "(unknown file)", 0, "(unknown function)"
        while hasattr(f, "f_code"):
            co = f.f_code
            filename = os.path.normcase(co.co_filename)
            if filename == logging._srcfile:
                f = f.f_back
                continue
            rv = (co.co_filename, f.f_lineno, co.co_name)
            break
        return rv

    rootLogger = logging.getLogger()
    isEnabled = rootLogger.isEnabledFor(level)
    d = _frame_stuff[0]
    try:
        logging.Logger.findCaller = findCaller
        _frame_stuff[0] = d + n
        yield isEnabled
    except:
        raise
    finally:
        logging.Logger.findCaller = _frame_stuff[1]
        _frame_stuff[0] = d


def A(x):
    with logdelta(1):
        logging.info('A: ' + x)    # Don't log with this line number

def B(x):
    with logdelta(2):
        logging.info('A: ' + x)    # or with this line number

def C(x):
    B(x)                       # or this line number

A('hello')                     # Instead, log with THIS line number
C('hello')                     # or THIS line number```

Keri