Python logging handler to append to list

Question

Somewhat similar to this (unanswered) question here:

https://stackoverflow.com/questions/33136398/python-logging-handler-that-appends-messages-to-a-list

I am looking to use a handler to simply append anything (passing a filter) to a python list. I would envisage some kind of code like this:

import logging
import sys
mylog = logging.getLogger()
mylog.setLevel(logging.DEBUG)

log_list = []
lh = logging.SomeHandler(log_list)
lh.setLevel(logging.DEBUG)

mylog.addHandler(lh)
mylog.warning('argh')
print log_list[0]

The question is therefore - how can I implement this? What SomeHandler can I use?

Extragalactic

the linked question has been removed from Stack Overflow for moderation reasons. Could you please remove the link from the question body, as it is no longer helpful? Thank you! – Lymphoma

Answer 1

Here is a naive, non thread-safe implementation:

import logging

class ListHandler(logging.Handler): # Inherit from logging.Handler
        def __init__(self, log_list):
                # run the regular Handler __init__
                logging.Handler.__init__(self)
                # Our custom argument
                self.log_list = log_list
        def emit(self, record):
                # record.message is the log message
                self.log_list.append(record.msg) 

Contrariety

This looks good on paper, but I get the error AttributeError: 'LogRecord' object has no attribute 'message' when trying to implement it (as per code in OP). Any ideas? – Extragalactic

OK - I figured it out. This could be version specific (I'm running 2.7.6), but the emit function needs to append record.msg, not record.message, ie: self.logs_list.append(record.msg) --if you could correct this @imriqwe, I'll mark it as the accepted answer. Cheers! – Extragalactic

Glad I could help :) – Contrariety

@Contrariety having a little difficulty putting this into action. My goal is simply to create a second handler that writes to the array when a message is logged. Can you expand on answer a bit? – Nanaam

Answer 2

@imriqwe's answer is correct for a non thread-safe implementation, but if you need to be thread-safe, one solution is to use a queue.Queue() instead of a list. Here is some code I am using in an in-process project to generate a tkinter log window.

import logging
import queue

class QueuingHandler(logging.Handler):
    """A thread safe logging.Handler that writes messages into a queue object.

       Designed to work with LoggingWidget so log messages from multiple
       threads can be shown together in a single ttk.Frame.

       The standard logging.QueueHandler/logging.QueueListener can not be used
       for this because the QueueListener runs in a private thread, not the
       main thread.

       Warning:  If multiple threads are writing into this Handler, all threads
       must be joined before calling logging.shutdown() or any other log
       destinations will be corrupted.
    """

    def __init__(self, *args, message_queue, **kwargs):
        """Initialize by copying the queue and sending everything else to superclass."""
        logging.Handler.__init__(self, *args, **kwargs)
        self.message_queue = message_queue

    def emit(self, record):
        """Add the formatted log message (sans newlines) to the queue."""
        self.message_queue.put(self.format(record).rstrip('\n'))

To use, create a queue, create the handler using the queue, then add it to the logger (this example also creates a log file in the current directory):

LOG_FORMAT = '%(asctime)s: %(name)8s: %(levelname)8s: %(message)s'
#  Setup root logger to write to a log file.
logging.basicConfig(filename='gui-test.log',
                    filemode='w',
                    format=LOG_FORMAT,
                    level=logging.DEBUG
                   )

#  Get a child logger
logger = logging.getLogger(name='gui')

#  Build our QueuingHandler
message_queue = queue.Queue()
handler = QueuingHandler(message_queue=message_queue, level=logging.DEBUG)

#  Change the date/time format for the GUI to drop the date
formatter = logging.Formatter(LOG_FORMAT)
formatter.default_time_format = '%H:%M:%S'
handler.setFormatter(formatter)

#  Add our QueuingHandler into the logging heirarchy at the lower level
logger.addHandler(handler)

Now all you have to do is read your messages from the queue.

Luis