Can Python's logging format be modified depending on the message log level?

Question

I'm using Python's logging mechanism to print output to the screen. I could do this with print statements, but I want to allow a finer-tuned granularity for the user to disable certain types of output. I like the format printed for errors, but would prefer a simpler format when the output level is "info."

For example:

  logger.error("Running cmd failed")
  logger.info("Running cmd passed")

In this example, I would like the format of the error to be printed differently:

# error
Aug 27, 2009 - ERROR: Running cmd failed
# info
Running cmd passed

Is it possible to have different formats for different log levels without having multiple logging objects? I'd prefer to do this without modifying the logger once it's created since there are a high number of if/else statements to determine how the output should be logged.

Cheat

Answer 1

Yes, you can do this by having a custom Formatter class:

class MyFormatter(logging.Formatter):
    def format(self, record):
        #compute s according to record.levelno
        #for example, by setting self._fmt
        #according to the levelno, then calling
        #the superclass to do the actual formatting
        return s

Then attach a MyFormatter instance to your handlers.

Paleolith

Outstanding - that worked perfectly. I modified the format() method to check levelno and change the message if need be. Otherwise it resets it back to the original string I passed in. Thanks! – Cheat

Please remove the checkmark from this answer. The one just below this is complete. There are wide swaths of code missing from this answer where there are just comments describing what you should do. – Guarnerius

Answer 2

I just ran into this issue and had trouble filling in the "holes" left in the above example. Here's a more complete, working version that I used. Hopefully this helps someone:

# Custom formatter
class MyFormatter(logging.Formatter):

    err_fmt  = "ERROR: %(msg)s"
    dbg_fmt  = "DBG: %(module)s: %(lineno)d: %(msg)s"
    info_fmt = "%(msg)s"


    def __init__(self, fmt="%(levelno)s: %(msg)s"):
        logging.Formatter.__init__(self, fmt)


    def format(self, record):

        # Save the original format configured by the user
        # when the logger formatter was instantiated
        format_orig = self._fmt

        # Replace the original format with one customized by logging level
        if record.levelno == logging.DEBUG:
            self._fmt = MyFormatter.dbg_fmt

        elif record.levelno == logging.INFO:
            self._fmt = MyFormatter.info_fmt

        elif record.levelno == logging.ERROR:
            self._fmt = MyFormatter.err_fmt

        # Call the original formatter class to do the grunt work
        result = logging.Formatter.format(self, record)

        # Restore the original format configured by the user
        self._fmt = format_orig

        return result

Edit:

Compliments of Halloleo, here's an example of how to use the above in your script:

fmt = MyFormatter()
hdlr = logging.StreamHandler(sys.stdout)

hdlr.setFormatter(fmt)
logging.root.addHandler(hdlr)
logging.root.setLevel(DEBUG)

Edit 2:

Python3 logging has changed a bit. See here for a Python3 approach.

Zamora

This works great! I changed the references to MyFormatter by name to self for more consistency. – Ohl

And here I might add the way you can use the MyFormatter class in your program (replace each <CR> with a carriage return): fmt = MyFormatter()<CR> hdlr = logging.StreamHandler(sys.stdout)<CR> <CR> hdlr.setFormatter(fmt)<CR> logging.root.addHandler(hdlr)<CR> logging.root.setLevel(DEBUG)`<CR> – Ohl

This answer won't work after 3.2, due to the changes in internal logging mechanism. logging.Formatter.format now depends of the style parameter of __init__. – Willable

Evpok is right. Add this after you assign self._fmt: self._style = logging.PercentStyle(self._fmt) – Chicory

Better use %(message) instead of %(msg), otherwise you can't use format strings (e.g. logger.info("foo %s", "bar")) – Eikon

Could this be improved by using super() instead of calling logging.Formatter? – Crichton

@phoenix: Yes. Good observation. – Zamora

@Willable your comment did save me a lot of effort. Cheers!! – Bowing

Using this, the line number that gets logged is the line number of the call to logger, not the line of the error in question. Any work around for this? – Sure

how can I provide time format here ? generally we format for time like this formatter = logging.Formatter("[%(asctime)s] -[%(levelname)] - %(message)s","%d-%m-%Y %I:%M:%S %p") – Northeastwards

Do you have an idea why the __init__ function is not called when I use a custom formatter with dictConfig and pass it via ()? – Excretory

Answer 3

One way of doing this

Define a class

import logging

class CustomFormatter(logging.Formatter):
    """Logging Formatter to add colors and count warning / errors"""

    grey = "\x1b[38;21m"
    yellow = "\x1b[33;21m"
    red = "\x1b[31;21m"
    bold_red = "\x1b[31;1m"
    reset = "\x1b[0m"
    format = "%(asctime)s - %(name)s - %(levelname)s - %(message)s (%(filename)s:%(lineno)d)"

    FORMATS = {
        logging.DEBUG: grey + format + reset,
        logging.INFO: grey + format + reset,
        logging.WARNING: yellow + format + reset,
        logging.ERROR: red + format + reset,
        logging.CRITICAL: bold_red + format + reset
    }

    def format(self, record):
        log_fmt = self.FORMATS.get(record.levelno)
        formatter = logging.Formatter(log_fmt)
        return formatter.format(record)

Instantiate logger

# create logger with 'spam_application'
logger = logging.getLogger("My_app")
logger.setLevel(logging.DEBUG)

# create console handler with a higher log level
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)

ch.setFormatter(CustomFormatter())

logger.addHandler(ch)

And use!

logger.debug("debug message")
logger.info("info message")
logger.warning("warning message")
logger.error("error message")
logger.critical("critical message")

Result

Ovum

This seems to be the only answer working on python 3.6 – Vagabond

I try to keep the same answer up to date here https://mcmap.net/q/25734/-how-can-i-color-python-logging-output, if helps please upvote. Thank you @Vagabond – Ovum

This is an excellent answer. Colour coding logging is also very helpful. – Clobber

Note that you can instantiate the formatters once in the init instead of on every call to format – Simulacrum

Answer 4

And again like JS answer but more compact.

class SpecialFormatter(logging.Formatter):
    FORMATS = {logging.DEBUG :"DBG: %(module)s: %(lineno)d: %(message)s",
               logging.ERROR : "ERROR: %(message)s",
               logging.INFO : "%(message)s",
               'DEFAULT' : "%(levelname)s: %(message)s"}

    def format(self, record):
        self._fmt = self.FORMATS.get(record.levelno, self.FORMATS['DEFAULT'])
        return logging.Formatter.format(self, record)

hdlr = logging.StreamHandler(sys.stderr)
hdlr.setFormatter(SpecialFormatter())
logging.root.addHandler(hdlr)
logging.root.setLevel(logging.INFO)

Polynices

This answer won't work after 3.2, due to the changes in internal logging mechanism. logging.Formatter.format now depends of the style parameter of __init__. – Willable

Answer 5

Instead of relying on styles or internal fields, you could also create a Formatter that delegates to other formatters depending on record.levelno (or other criteria). This is a slightly cleaner solution in my humble opinion. Code below should work for any python version >= 2.7:

The simple way would look something like this:

class MyFormatter(logging.Formatter):

    default_fmt = logging.Formatter('%(levelname)s in %(name)s: %(message)s')
    info_fmt = logging.Formatter('%(message)s')

    def format(self, record):
        if record.levelno == logging.INFO:
            return self.info_fmt.format(record)
        else:
            return self.default_fmt.format(record)

But you could make it more generic:

class VarFormatter(logging.Formatter):

    default_formatter = logging.Formatter('%(levelname)s in %(name)s: %(message)s')

    def __init__(self, formats):
        """ formats is a dict { loglevel : logformat } """
        self.formatters = {}
        for loglevel in formats:
            self.formatters[loglevel] = logging.Formatter(formats[loglevel])

    def format(self, record):
        formatter = self.formatters.get(record.levelno, self.default_formatter)
        return formatter.format(record)

I used a dict as input here, but obviously you could also use tuples, **kwargs, whatever floats your boat. This would then be used like:

formatter = VarFormatter({logging.INFO: '[%(message)s]', 
                          logging.WARNING: 'warning: %(message)s'})
<... attach formatter to logger ...>

Experience

Answer 6

This is an adaptation of estani's answer to the new implementation of logging.Formatter which now relies on formatting styles. Mine relies on '{' style format, but it can be adapted. Could be refined to be more general and allow selection of formatting style and custom messages as arguments to __init__, too.

class SpecialFormatter(logging.Formatter):
    FORMATS = {logging.DEBUG : logging._STYLES['{']("{module} DEBUG: {lineno}: {message}"),
           logging.ERROR : logging._STYLES['{']("{module} ERROR: {message}"),
           logging.INFO : logging._STYLES['{']("{module}: {message}"),
           'DEFAULT' : logging._STYLES['{']("{module}: {message}")}

    def format(self, record):
        # Ugly. Should be better
        self._style = self.FORMATS.get(record.levelno, self.FORMATS['DEFAULT'])
        return logging.Formatter.format(self, record)

hdlr = logging.StreamHandler(sys.stderr)
hdlr.setFormatter(SpecialFormatter())
logging.root.addHandler(hdlr)
logging.root.setLevel(logging.INFO)

Willable

Thanks for updating this to work with Python3. I ran into the same problem in Python3 and came up with a similar solution. Would you please be so kind as to post this answer there as well? #14845470 – Zamora

Using the new '{' style in your comments? :-) – Zamora

Answer 7

The above solution works with 3.3.3 release. However with 3.3.4 you get the following error.

FORMATS = { logging.DEBUG : logging._STYLES['{']("{module} DEBUG: {lineno}: {message}"),

TypeError: 'tuple' object is not callable

After some searching around in the logging class Lib\logging__init__.py I found that a data structure has changed from 3.3.3 to 3.3.4 that causes the issue

3.3.3

_STYLES = {
    '%': PercentStyle,
    '{': StrFormatStyle,
    '$': StringTemplateStyle
}

3.3.4

_STYLES = {
   '%': (PercentStyle, BASIC_FORMAT),
   '{': (StrFormatStyle, '{levelname}:{name}:{message} AA'),
    '$': (StringTemplateStyle, '${levelname}:${name}:${message} BB'),
}

The updated solution is therefore

class SpecialFormatter(logging.Formatter):
     FORMATS = {logging.DEBUG : logging._STYLES['{'][0]("{module} DEBUG: {lineno}: {message}"),
       logging.ERROR : logging._STYLES['{'][0]("{module} ERROR: {message}"),
       logging.INFO : logging._STYLES['{'][0]("{module}: {message}"),
       'DEFAULT' : logging._STYLES['{'][0]("{module}: {message}")}

 def format(self, record):
    # Ugly. Should be better
    self._style = self.FORMATS.get(record.levelno, self.FORMATS['DEFAULT'])
    return logging.Formatter.format(self, record)

Odom

It may be easier to just import the style type directly with from logging import StrFormatStyle instead of doing logging._STYLES['{'][0] – Europium

Answer 8

If you are just looking to skip formatting certain levels, you can do something simpler than the other answers like the following:

class FormatterNotFormattingInfo(logging.Formatter):
    def __init__(self, fmt = '%(levelname)s:%(message)s'):
        logging.Formatter.__init__(self, fmt)

    def format(self, record):
        if record.levelno == logging.INFO:
            return record.getMessage()
        return logging.Formatter.format(self, record)

This also has the advantage of working before and after the 3.2 release by not using internal variables like self._fmt nor self._style.

Willemstad

I think this is most clean solution – Abey