Python rolling log to a variable

Question

I have an application that makes use of multi-threading and is run in the background on a server. In order to monitor the application without having to log on to the server, I decided to include Bottle in order to respond to a few HTTP endpoints and report status, perform remote shutdown, etc.

I also wanted to add a way to consult the logfile. I could log using the FileHandler and send the destination file when the URL is requested (e.g. /log).

However, I was wondering if it'd be possible to implement something like a RotatingFileHandler, but instead of logging to a file, logging to a variable (e.g. BytesIO). This way, I could limit the log to the most recent information, while at the same time being able to return it to the browser as text instead of as a separate file download.

The RotatingFileHandler requires a filename, so it's not an option to pass it a BytesIO stream. Logging to a variable itself is perfectly doable (e.g. Capturing Python Log Output In A Variable), but I'm a bit stumped on how to do the rolling part.

Any thoughts, hints, suggestions would be greatly appreciated.

Reginareginald

Could you use a collections.deque object to capture the log output? You would set a maximum length for the deque, and once it reaches this length old log items pop off the start as you append new log items. – Trample

Answer 1

Going further on Andrew Guy's suggestion, I subclassed logging.Handler and implemented a handler that uses collections.deque with a fixed length to keep a record of the log messages.

import logging
import collections


class TailLogHandler(logging.Handler):

    def __init__(self, log_queue):
        logging.Handler.__init__(self)
        self.log_queue = log_queue

    def emit(self, record):
        self.log_queue.append(self.format(record))


class TailLogger(object):

    def __init__(self, maxlen):
        self._log_queue = collections.deque(maxlen=maxlen)
        self._log_handler = TailLogHandler(self._log_queue)

    def contents(self):
        return '\n'.join(self._log_queue)

    @property
    def log_handler(self):
        return self._log_handler

Example usage:

import random

logger = logging.getLogger(__name__)

tail = TailLogger(10)

formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

log_handler = tail.log_handler
log_handler.setFormatter(formatter)
logger.addHandler(log_handler)

levels = [logging.INFO, logging.ERROR, logging.WARN, logging.DEBUG, logging.CRITICAL]
logger.setLevel(logging.ERROR)

for i in range(500):
    logger.log(random.choice(levels), 'Message {}'.format(i))

print(tail.contents())

Output:

2016-06-22 13:58:25,975 - __main__ - CRITICAL - Message 471
2016-06-22 13:58:25,975 - __main__ - ERROR - Message 472
2016-06-22 13:58:25,975 - __main__ - ERROR - Message 473
2016-06-22 13:58:25,975 - __main__ - ERROR - Message 474
2016-06-22 13:58:25,975 - __main__ - ERROR - Message 477
2016-06-22 13:58:25,975 - __main__ - CRITICAL - Message 481
2016-06-22 13:58:25,975 - __main__ - CRITICAL - Message 483
2016-06-22 13:58:25,975 - __main__ - ERROR - Message 484
2016-06-22 13:58:25,975 - __main__ - CRITICAL - Message 485
2016-06-22 13:58:25,976 - __main__ - CRITICAL - Message 490

Reginareginald

Answer 2

Use the technique described in Capturing Python Log Output In A Variable, but capture it into a custom stream that throws away old data.

Like so:

# Adapted from http://alanwsmith.com/capturing-python-log-output-in-a-variable

import logging
import io
import collections

class FIFOIO(io.TextIOBase):
    def __init__(self, size, *args):
        self.maxsize = size
        io.TextIOBase.__init__(self, *args)
        self.deque = collections.deque()
    def getvalue(self):
        return ''.join(self.deque)
    def write(self, x):
        self.deque.append(x)
        self.shrink()
    def shrink(self):
        if self.maxsize is None:
            return
        size = sum(len(x) for x in self.deque)
        while size > self.maxsize:
            x = self.deque.popleft()
            size -= len(x)

### Create the logger
logger = logging.getLogger('basic_logger')
logger.setLevel(logging.DEBUG)

### Setup the console handler with a FIFOIO object
log_capture_string = FIFOIO(256)
ch = logging.StreamHandler(log_capture_string)
ch.setLevel(logging.DEBUG)

### Optionally add a formatter
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
ch.setFormatter(formatter)

### Add the console handler to the logger
logger.addHandler(ch)


### Send log messages. 
logger.debug('debug message')
logger.info('info message')
logger.warn('warn message')
logger.error('error message')
logger.critical('critical message')


### Pull the contents back into a string and close the stream
log_contents = log_capture_string.getvalue()
log_capture_string.close()

### Output as lower case to prove it worked. 
print(log_contents.lower())

Repudiation

Thank you for you input. The only worry I have with this implementation (this wasn't specified in the original post), is that the docs state that "TextIOWrapper objects are not thread-safe", but they don't say anything about TextIOBase` objects. Also, the same docs say that StringIO should be significantly faster than TextIO when dealing with large files. Could you just swap TextIOBase with StringIO as your FIFOIO base class? – Iosep