python log formatter that shows all kwargs in extra

Question

I hope to add the following log points to my application and display the full contents of extra on console, e.g.,

logger.info('Status', extra={'foo':data})
logger.info('Status', extra={'bar':data})
logger.info('Status', extra={'foo':data, 'bar':data}) 

and I hope to see:

2016-10-10 15:28:31,408, INFO, Status, foo=data
2016-10-10 15:38:31,408, INFO, Status, bar=data
2016-10-10 15:48:31,408, INFO, Status, foo=data, bar=data

Is this even possible? According to official logging documentation, the Formatter must be set up with a format string that expects foo and bar but in my case all I want is to dump out the entire kwargs of extra without prior knowledge of foo and bar.

Eggplant

Take a look at bugs.python.org/file4410/logging.py. I don't have time to work out a full example, but I think you could abuse the args attr on the LogRecord and just pass in a dict. – Lecherous

Answer 1

Trying to solve the same problem right now. The problem is that anything passed to extra is added as a property of LogRecord and the formatter cannot distinguish it from other LogRecord properties. I only came up with a hackish approach of creating a dummy record and comparing it with the actual one:

class ExtraLogFormatter(logging.Formatter):                                                                             
    def format(self, record):                                                                                           
        dummy = logging.LogRecord(None,None,None,None,None,None,None)                                                   
        extra_txt = ''                                                                                                  
        for k,v in record.__dict__.items():                                                                             
            if k not in dummy.__dict__:                                                                                 
                extra_txt += ', {}={}'.format(k,v)                                                                      
        message = super().format(record)                                                                                
        return message + extra_txt 

Inly

Answer 2

Entries of the object passed as extra will become member variables of the LogRecord.

logger.info('Status', extra={'my_params':{'foo':3, 'bar':4}})

Then you can format it as:

handler.setFormatter(logging.Formatter('%(message)s %(my_params)s'))

Which will result in the following log:

#> Status {'foo':3, 'bar':4}}

This assumes you have control over the 'extra' parameter at the time of logging.

Culler

Answer 3

One option would be to define a simple utility function:

def extra(**kwargs):
    return {"extra": kwargs}

Then your usage:

logger.info('Status', extra={'foo':data, 'bar':data}) 

Becomes not too different:

logger.info('Status', **extra(foo=data, bar=data)) 

And you can set the formatting easily:

handler.setFormatter(logging.Formatter('%(message)s %(extra)s'))

Fanfaron