Python's ftplib with tqdm

Question

I have a console script which uses ftplib as a backend to get a number of files from an ftp server. I would like to use tqdm to give the user some feedback provided they have a "verbose" switch on. This must be optional as some users might use the script without tty access.

The ftplib's retrbinary method takes a callback so it should be possible to hook tqdm in there somehow. However, I have no idea what this callback would look like.

Bangup

Answer 1

From FTP.retrbinary:

The callback function is called for each block of data received, with a single string argument giving the data block.

So the callback could be something like:

with open(filename, 'wb') as fd:
    total = ftpclient.size(filename)

    with tqdm(total=total) as pbar:
        def callback_(data):
            l = len(data)
            pbar.update(l)
            fd.write(data)

        ftpclient.retrbinary('RETR {}'.format(filename), callback_)

Beware: This code is untested and probably has to be adapted.

Durden

After implementing this, it needs a few more checks to handle bad blocks. if (raw_file.tell() == filesize): tqdm_instance.close() – Hebrides

Answer 2

That code shouldn't work as pbar will be "closed" when the with block terminates, which occurs just before ftpclient.retrbinary(...). You need a very minor indentation mod:

with open(filename, 'wb') as fd:
    total = ftpclient.size(filename)

    with tqdm(total=total,
              unit='B', unit_scale=True, unit_divisor=1024, 
              disable=not verbose) as pbar:
        def cb(data):
            pbar.update(len(data))
            fd.write(data)

        ftpclient.retrbinary('RETR {}'.format(filename), cb)

EDIT added disable flag and bytes scaling

Singly

Answer 3

with open(filename, 'wb') as fd:
    total = ftpclient.size(filename)

    with tqdm(total=total,
              unit='B', unit_scale=True, unit_divisor=1024, 
              disable=not verbose) as pbar:
        def cb(data):
            pbar.update(len(data))
            fd.write(data)

        ftpclient.retrbinary('RETR {}'.format(filename), cb)

Photoemission