Python progress bar through logging module

Question

I have seen different solutions for a progress bar within Python, but the simple stdout solutions are not working for my project. I have multiple classes and use the "logging" module to output information to stdout. I have a function of which I want to show a progress bar on one line, flushing the buffer each time.

Example of the simple progress:

for i in range(100):
    time.sleep(1)
    sys.stdout.write("\r%d%%" %i)
    sys.stdout.flush()

When I try to write via stdout and then flush the buffer, either the buffer is not flushed or the progress doesn't go anywhere. I am hoping to avoid some sort of threading or complicated process to make this possible. Does someone have a preferred way of making this happen?

Dichlamydeous

Does this answer your question? Change logging "print" function to "tqdm.write" so logging doesn't interfere with progress bars – Radioelement

Answer 1

I couldn't find a good solution for this, so I wrote enlighten progress bar to handle it. Basically it changes the scroll area of the terminal so logging is done above the progress bar(s) rather than having to redraw the progress bar(s) every time you want to write to STDOUT. This lets you write to the terminal as much as you want without having to modify logging, print, etc.

import logging
import time
import enlighten

# Setup logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger()

# Setup progress bar
manager = enlighten.get_manager()
pbar = manager.counter(total=100, desc='Ticks', unit='ticks')

for i in range(1, 101):
    logger.info("Processing step %s" % i)
    time.sleep(.2)
    pbar.update()

Vireo

I just discovered this module, exactly what I needed! Thank you for this. – Gottfried

Removed comment about Windows. Windows support was added in version 1.3.0 – Vireo

Version 1.10.0 added support for Jupyter Notebooks – Vireo

Answer 2

I solved it like this:

import logging
import time
from tqdm import tqdm
import io

class TqdmToLogger(io.StringIO):
    """
        Output stream for TQDM which will output to logger module instead of
        the StdOut.
    """
    logger = None
    level = None
    buf = ''
    def __init__(self,logger,level=None):
        super(TqdmToLogger, self).__init__()
        self.logger = logger
        self.level = level or logging.INFO
    def write(self,buf):
        self.buf = buf.strip('\r\n\t ')
    def flush(self):
        self.logger.log(self.level, self.buf)

if __name__ == "__main__":
    logging.basicConfig(format='%(asctime)s [%(levelname)-8s] %(message)s')
    logger = logging.getLogger()
    logger.setLevel(logging.DEBUG)

    tqdm_out = TqdmToLogger(logger,level=logging.INFO)
    for x in tqdm(range(100),file=tqdm_out,mininterval=30,):
        time.sleep(.5)

Output

2016-12-19 15:35:06 [INFO    ] 16%|#####9                                | 768/4928 [07:04<40:50,  1.70it/s]
2016-12-19 15:36:07 [INFO    ] 18%|######6                               | 865/4928 [08:04<40:34,  1.67it/s]

Ship

Just curious where does the number in the end of the progress bar #####9 come from, and how can we get rid of it? – Pretorius

This is nice, but it still prints a new line for each update, or? – Tolbooth

All numbers are 'decimals'. And indeed, logger will always print on a new line. Which is nice if you log to a file. If you don't want a new line (with timestamp), then you don't need to use logger. – Ship

I like that this solution sends the tqdm output through the Python logger, which is what I want to do. Most of the questions/solutions I find related to tqdm and logging is to send all other logging through tqdm's output handler. – Morphophoneme

Answer 3

You can use the tqdm progress bar with a custom handler through logging as described here:

import logging
import time
import colorlog
from tqdm import tqdm

class TqdmHandler(logging.StreamHandler):
    def __init__(self):
        logging.StreamHandler.__init__(self)

    def emit(self, record):
        msg = self.format(record)
        tqdm.write(msg)

if __name__ == "__main__":
    for x in tqdm(range(100)):
        logger = colorlog.getLogger("MYAPP")
        logger.setLevel(logging.DEBUG)
        handler = TqdmHandler()
        handler.setFormatter(colorlog.ColoredFormatter(
            '%(log_color)s%(name)s | %(asctime)s | %(levelname)s | %(message)s',
            datefmt='%Y-%d-%d %H:%M:%S',
            log_colors={
                'DEBUG': 'cyan',
                'INFO': 'white',
                'SUCCESS:': 'green',
                'WARNING': 'yellow',
                'ERROR': 'red',
                'CRITICAL': 'red,bg_white'},))

        logger.addHandler(handler)
        logger.debug("Inside subtask: "+str(x))
        time.sleep(.5)

Faddish

It doesn't work for me. I think, you can try this: gist.github.com/w495/80f6a7351a10d5b6cecb5dad6c8cd8d6 – Ladoga

should datefmt='%Y-%d-%d %H:%M:%S' be like: datefmt='%Y-%m-%d %H:%M:%S' (month)? – Procarp

Answer 4

If you know the progress bar is always going to be written to STDOUT, you should just be using print instead of a logger. See the documentation in the Python logging tutorial documentation

Whetstone

The problem with using "print" is that it throws each value on a new line no longer making the output a progress bar. STDOUT is supposed to flush the buffer and write over the existing line, but this does not appear to work when using it in the code that is also logging. Mind you, I am not wrapping any logging module syntax around the progress bar code. – Dichlamydeous

Using the exact code you posted above, I see the output being replaced in place. Are you seeing each number on a new line? – Whetstone

does this answer the question? – Dogwood

Answer 5

You might want to use progressbar2 which allows to keep logging while having the progressbar printed:

Minimal example:

import time
import progressbar

for i in progressbar.progressbar(range(100), redirect_stdout=True):
    print('Some text', i)
    time.sleep(0.1)

Will result in:

(progressbar2 is compatible with both Python2 and Python3)

Aeschines

Answer 6

Cleaning-up the code on these proposals, this is IMO the correct implementation, with no external dependencies other than tqdm (also posted here):

import logging
from tqdm import tqdm


class TqdmLoggingHandler(logging.StreamHandler):
    """Avoid tqdm progress bar interruption by logger's output to console"""
    # see logging.StreamHandler.eval method:
    # https://github.com/python/cpython/blob/d2e2534751fd675c4d5d3adc208bf4fc984da7bf/Lib/logging/__init__.py#L1082-L1091
    # and tqdm.write method:
    # https://github.com/tqdm/tqdm/blob/f86104a1f30c38e6f80bfd8fb16d5fcde1e7749f/tqdm/std.py#L614-L620

    def emit(self, record):
        try:
            msg = self.format(record)
            tqdm.write(msg, end=self.terminator)
        except RecursionError:
            raise
        except Exception:
            self.handleError(record)

Testing:

import time

log = logging.getLogger(__name__)
log.setLevel(logging.INFO)
log.addHandler(TqdmLoggingHandler())
#   ^-- Assumes this will be the unique handler emitting messages to sys.stdout.
#       If other handlers output to sys.stdout (without tqdm.write),
#       progress bar will be interrupted by those outputs

for i in tqdm(range(20)):
    log.info(f"Looping {i}")
    time.sleep(0.1)

Izzard