How do i monitor the progress of a file transfer through pysftp

Question

I'm using python 3.3.2 and pysftp to make a back-up utility that stores copies of files on another computer on my network.

I already know how to use pysftp to transfer theses files however i would like to see the progress of the transfer (bytes per second, percent completion, time remaining) instead of some static print statement indicating that the transfer has started or finished

something like this maybe ( filename: 3.4MiB/s | 40%<#########=============> | 0:03:54 remaining )

Edit: I could probably build a progress bar if i just knew how to get pysftp's put command to yeild the information, THAT is what i want to know how to do

Edit: I think i found my answer on another question after extensive digging:

How to see (log) file transfer progress using paramiko?

Jacobah

Answer 1

There exists a call back function in both get/put methods (see the official documentation)

get(remotepath, localpath=None, callback=None, preserve_mtime=False)

where callback can look like:

lambda x,y: print("{} transfered out of {}".format(x,y))

Ietta

Answer 2

You can create a function which prints the transfer every (lets say) 10 percent:

progressDict={}
progressEveryPercent=10

for i in range(0,101):
    if i%progressEveryPercent==0:
        progressDict[str(i)]=""

def printProgressDecimal(x,y):
    if int(100*(int(x)/int(y))) % progressEveryPercent ==0 and progressDict[str(int(100*(int(x)/int(y))))]=="":
        print("{}% ({} Transfered(B)/ {} Total File Size(B))".format(str("%.2f" %(100*(int(x)/int(y)))),x,y))
        progressDict[str(int(100*(int(x)/int(y))))]="1"

Then you can call that function in your get or put command like this:

sftp.get(eachFile,localpath=localDirectory+eachFile, callback=lambda x,y: printProgressDecimal(x,y))

A Sample Output Is:

0.00% (32768 Transfered(B)/ 1108907068 Total File Size(B))
10.00% (110919680 Transfered(B)/ 1108907068 Total File Size(B))
20.00% (221806592 Transfered(B)/ 1108907068 Total File Size(B))
30.00% (332693504 Transfered(B)/ 1108907068 Total File Size(B))
40.00% (443580416 Transfered(B)/ 1108907068 Total File Size(B))
50.00% (554467328 Transfered(B)/ 1108907068 Total File Size(B))
60.00% (665354240 Transfered(B)/ 1108907068 Total File Size(B))
70.00% (776241152 Transfered(B)/ 1108907068 Total File Size(B))
80.00% (887128064 Transfered(B)/ 1108907068 Total File Size(B))
90.00% (998047744 Transfered(B)/ 1108907068 Total File Size(B))
100.00% (1108907068 Transfered(B)/ 1108907068 Total File Size(B))

Elayneelazaro

Answer 3

import math, pysftp

with pysftp.Connection(host, username, password) as sftp:
    sftp.get(remotepath, localpath, callback=lambda x,y: progressbar(x,y))

def progressbar(x, y):
    ''' progressbar for the pysftp
    '''
    bar_len = 60
    filled_len = math.ceil(bar_len * x / float(y))
    percents = math.ceil(100.0 * x / float(y))
    bar = '=' * filled_len + '-' * (bar_len - filled_len)
    filesize = f'{math.ceil(y/1024):,} KB' if y > 1024 else f'{y} byte'
    sys.stdout.write(f'[{bar}] {percents}% {filesize}\r')
    sys.stdout.flush()

[============================================================] 100% 4,342 KB

Overbuild

Answer 4

If you want to use the tqdm progressbar:

from tqdm import tqdm
import pysftp

class Progressbar(tqdm):
    def update_to(self, gathered: int, total: int):
        self.total = total
        self.update(gathered - self.n)

pb = Progressbar(
            desc=f"downloading {filename}",
            unit="ib",
            unit_divisor=1 << 20, #MiB
            unit_scale=True,
        )

with pysftp.Connection(host, username, password) as sftp:
    sftp.get(filename, callback=pb.update_to)

Loveridge

Answer 5

This is what it would look like using rich.

import pysftp
from rich.progress import Progress

with pysftp.Connection(host=None, username=None, password=None) as sftp:
    file_to_download = '.'
    with Progress() as progress:
        task = progress.add_task(f"[red]Downloading {file_to_download} ...", total=None)

        sftp.get(
            file_to_download,
            local_path,
            callback=lambda so_far, total: progress.update(task, completed=so_far, total=total),
        )

Pinelli

Answer 6

You can either write your own progressbar, which is fairly simple if you know anything about terminals. Or you can use progressbar. Here's a simple example from the documentation:

@example
def example1():
    widgets = ['Test: ', Percentage(), ' ', Bar(marker=RotatingMarker()),
               ' ', ETA(), ' ', FileTransferSpeed()]
    pbar = ProgressBar(widgets=widgets, maxval=10000000).start()
    for i in range(1000000):
        # do something
        pbar.update(10*i+1)
    pbar.finish()

In both cases, you will need the file transfer method to yield something during the transfer. If you can get it to yield the bytes it receives, it's very easy to create a progress bar either way.

Causation

Doesn't answer the user's question: How can we do this in pysftp? – Salesin