what is the difference between cmd and idle when using tqdm?

Question

recently I want to add a simple progress bar to my script, I use tqdm to that, but what puzzle me is that the output is different when I am in the IDLE or in the cmd

for example this

from tqdm import tqdm
import time

def test():
    for i in tqdm( range(100) ):
        time.sleep(0.1)

give the expected output in the cmd

30%|███       | 30/100 [00:03<00:07,  9.14it/s]

but in the IDLE the output is like this

  0%|          | 0/100 [00:00<?, ?it/s]
  1%|1         | 1/100 [00:00<00:10,  9.14it/s]
  2%|2         | 2/100 [00:00<00:11,  8.77it/s]
  3%|3         | 3/100 [00:00<00:11,  8.52it/s]
  4%|4         | 4/100 [00:00<00:11,  8.36it/s]
  5%|5         | 5/100 [00:00<00:11,  8.25it/s]
  6%|6         | 6/100 [00:00<00:11,  8.17it/s]
  7%|7         | 7/100 [00:00<00:11,  8.12it/s]
  8%|8         | 8/100 [00:00<00:11,  8.08it/s]
  9%|9         | 9/100 [00:01<00:11,  8.06it/s]
 10%|#         | 10/100 [00:01<00:11,  8.04it/s]
 11%|#1        | 11/100 [00:01<00:11,  8.03it/s]
 12%|#2        | 12/100 [00:01<00:10,  8.02it/s]
 13%|#3        | 13/100 [00:01<00:10,  8.01it/s]
 14%|#4        | 14/100 [00:01<00:10,  8.01it/s]
 15%|#5        | 15/100 [00:01<00:10,  8.01it/s]
 16%|#6        | 16/100 [00:01<00:10,  8.00it/s]
 17%|#7        | 17/100 [00:02<00:10,  8.00it/s]
 18%|#8        | 18/100 [00:02<00:10,  8.00it/s]
 19%|#9        | 19/100 [00:02<00:10,  8.00it/s]
 20%|##        | 20/100 [00:02<00:09,  8.00it/s]
 21%|##1       | 21/100 [00:02<00:09,  8.00it/s]
 22%|##2       | 22/100 [00:02<00:09,  8.00it/s]
 23%|##3       | 23/100 [00:02<00:09,  8.00it/s]
 24%|##4       | 24/100 [00:02<00:09,  8.00it/s]
 25%|##5       | 25/100 [00:03<00:09,  8.00it/s]
 26%|##6       | 26/100 [00:03<00:09,  8.00it/s]
 27%|##7       | 27/100 [00:03<00:09,  8.09it/s]
 28%|##8       | 28/100 [00:03<00:09,  7.77it/s]
 29%|##9       | 29/100 [00:03<00:09,  7.84it/s]
 30%|###       | 30/100 [00:03<00:08,  7.89it/s]
 31%|###1      | 31/100 [00:03<00:08,  7.92it/s]
 32%|###2      | 32/100 [00:03<00:08,  7.94it/s]
 33%|###3      | 33/100 [00:04<00:08,  7.96it/s]
 34%|###4      | 34/100 [00:04<00:08,  7.97it/s]
 35%|###5      | 35/100 [00:04<00:08,  7.98it/s]
 36%|###6      | 36/100 [00:04<00:08,  7.99it/s]
 37%|###7      | 37/100 [00:04<00:07,  7.99it/s]
 38%|###8      | 38/100 [00:04<00:07,  7.99it/s]
 39%|###9      | 39/100 [00:04<00:07,  8.00it/s]
 40%|####      | 40/100 [00:04<00:07,  8.00it/s]
 41%|####1     | 41/100 [00:05<00:07,  8.00it/s]

I also get the same result if I make my own progress bar like

import sys

def progress_bar_cmd(count,total,suffix="",*,bar_len=60,file=sys.stdout):
    filled_len = round(bar_len*count/total)
    percents   = round(100*count/total,2)
    bar        = "#"*filled_len + "-"*(bar_len - filled_len)
    file.write( "[%s] %s%s ...%s\r"%(bar,percents,"%",suffix))
    file.flush()

for i in range(101):
    time.sleep(1)
    progress_bar_cmd(i,100,"range 100") 

why is that????

and there is a way to fix it???

Pius

related: bugs.python.org/issue23220 – Gossoon

Answer 1

Limiting ourselves to ascii characters, the program output of your second code is the same in both cases -- a stream of ascii bytes representing ascii chars. The language definition does not and cannot specify what an output device or display program will do with the bytes, in particular with control characters such as '\r'.

The Windows Command Prompt console at least sometimes interprets '\r' as 'return the cursor to the beginning of the current line without erasing anything'. In a Win10 console:

>>> import sys; out=sys.stdout
>>> out.write('abc\rdef')
def7

However, when I run your second code, with the missing time import added, I do not see the overwrite behavior, but see the same continued line output as with IDLE.

C:\Users\Terry>python f:/python/mypy/tem.py
[------------------------------------------------------------] 0.0% ...range 100[#-----------------------------------------------------------] ...

On the third hand, if shorten the write to file.write("[%s]\r"% bar), then I do see one output overwritten over and over.

The tk Text widget used by IDLE only interprets \t and \n, but not other control characters. To some of us, this seems appropriate for a development environment, where erasing characters is less appropriate than in a production environment.

Schenck

so, in resumen to get the desire result I have to use the cmd? or there is a way to tell the idle to handle the others control characters? – Pius

You have to use the command window. I might add another option in the future to make testing easier. But it is not my highest priority. – Schenck

ok, then I guess that I should make a command line interface to my script and in the idle I will use tqdm_gui instead – Pius

A few months ago I added section `IDLE - console differences' to IDLE Help (on the IDLE Help menu as part of bugs.python.org/issue23220. I plan to revise it more. – Schenck

I should add that if you create your own user interface with tkinter, overwriting should be trivial. For a progress bar would use a Label with a StringVar textvariable. If the latter were called progress, then progress.set(new_value) would update the label to show new_value. On would use root.after instead of time.sleep to set the update interval, and allow other activity during that interval. – Schenck

mmm, I think that tkinter have a widgets for that, I experimented with it a long time ago but I have forgotten. My script is nothing fancy, just to download some img with urllib.request.urlretrieve so it don't really need a gui but maybe later I try using that to make a double progress bar and see if look better that the one from tqdm_gui. other than that, the one from tqdm is perfect, I only need to remember to execute the script in the cmd – Pius

tkinter.ttk.Progressbar is what you are thinking of. The point of my previous comment is that the contents of displayed text in tk widgets can be changed with widget calls, just not with \r. – Schenck