Dynamic refresh printing of multiprocessing or multithreading in Python

Question

I have implemented a multiprocessing downloader. How can I print the status bar (complete rate, download speed) which can refresh automatically in different part on the terminal.

Like this:

    499712  [6.79%]   68k/s     // keep refreshing
    122712  [16.79%]   42k/s    // different process/thread
     99712  [56.32%]   10k/s

code:

download(...)
...
    f = open(tmp_file_path, 'wb')
    print "Downloading: %s Bytes: %s" % (self.file_name, self.file_size)
    file_size_dl = 0
    block_sz = 8192
    start_time = time.time()
    while True:
        buffer = self.opening.read(block_sz)
        if not buffer:
            break

        file_size_dl += len(buffer)
        f.write(buffer)
        end_time = time.time()
        cost_time = end_time - start_time
        if cost_time == 0:
            cost_time = 1
        status = "\r%10d  [%3.2f%%]  %3dk/s" % (file_size_dl,
                file_size_dl * 100. / self.file_size,
                file_size_dl * 100. / 1024 / 1024 / cost_time)
        print status,
        sys.stdout.flush()
    f.close()

DownloadProcess inherits Process class and trigger the download method.

I use queue to store the url. Here is starting process

  ...
  for i in range(3):
    t = DownloadProcess(queue)
    t.start()
    for url in urls:
        queue.put(url)
  queue.join()

Reclinate

Could you as well provide come code for us to improve? – Paulapauldron

excess.org/urwid might help you, but only in terms of giving you nice positionable widgets for the task. You'll need to have a 'head' that manages all the lists you're displaying running in the master process. It'd most likely be easier to do a webpage tho. – Extrauterine

@Extrauterine I don't want to use a full functional library. I just need to give a simple display. – Reclinate

Answer 1

Below is a demo that has implemented both multi-processing and multi-threading. To try one or the other just uncomment the import lines at the top of the code. If you have a progress bar on a single line then you can use the technique that you have of printing '\r' to move the cursor back to the start of the line. But if you want to have multi-line progress bars then you are going to have to get a little fancier. I just cleared the screen each time I wanted to print the progress bars. Check out the article console output on Unix in Python it helped me a great deal in producing the code below. He shows both techniques. You can also give the curses library that is part of python standard library a shot. The question Multiline progress bars asks a similar thing. The main thread/process spawns the child threads that do the work and communicate their progress back to the main thread using a queue. I highly recommend using queues for inter-process/thread communication. The main thread then displays the progress and waits for all children to end execution before exiting itself.

code

import time, random, sys, collections
from multiprocessing import Process as Task, Queue
#from threading import Thread as Task
#from Queue import Queue

def download(status, filename):
    count = random.randint(5, 30)
    for i in range(count):
        status.put([filename, (i+1.0)/count])
        time.sleep(0.1)

def print_progress(progress):
    sys.stdout.write('\033[2J\033[H') #clear screen
    for filename, percent in progress.items():
        bar = ('=' * int(percent * 20)).ljust(20)
        percent = int(percent * 100)
        sys.stdout.write("%s [%s] %s%%\n" % (filename, bar, percent))
    sys.stdout.flush()

def main():
    status = Queue()
    progress = collections.OrderedDict()
    workers = []
    for filename in ['test1.txt', 'test2.txt', 'test3.txt']:
        child = Task(target=download, args=(status, filename))
        child.start()
        workers.append(child)
        progress[filename] = 0.0
    while any(i.is_alive() for i in workers):
        time.sleep(0.1)
        while not status.empty():
            filename, percent = status.get()
            progress[filename] = percent
            print_progress(progress)
    print 'all downloads complete'

main()

demo

Derain

Absolutely brilliant. Much simpler than the monstrosity I was imagining. Thanks. – Funnel

@Funnel That turns out to almost always be the case with Python :) – Mccain