How to see (log) file transfer progress using paramiko?

Question

I'm using Paramiko's SFTPClient to transfer file between hosts. I want my script to print the file transfer progress similar to the output seen using scp.

$ scp my_file user@host

user@host password: 

my_file                          100%  816KB 815.8KB/s   00:00

$

Any idea?

Thanks in advance

Clothe

Answer 1

Use the optional callback parameter of the put function. Something like this:

def printTotals(transferred, toBeTransferred):
    print "Transferred: {0}\tOut of: {1}".format(transferred, toBeTransferred)

sftp.put("myfile","myRemoteFile",callback=printTotals)

Lovelady

"Out of" may be a better term than "Still to send" since the last amount doesn't change. – Uxorial

@dss is that the case? It's been a couple years, and I don't remember if the toBeTransferred number is the total or the remainder. – Lovelady

@SpencerRathbun Yea seems to be. This is my print Transferred: 205979648 Out of: 263659867 Transferred: 206012416 Out of: 263659867 Transferred: 206045184 Out of: 263659867 Transferred: 206077952 Out of: 263659867 Transferred: 206110720 Out of: 263659867 Transferred: 206143488 Out of: 263659867 Transferred: 206176256 Out of: 263659867 The second number never changes so I changed it to "Out of:" – Uxorial

Works quite well. I combined the above answer with yours such that I can see the live status of bytes transferred (I chose every 100MB). – Kolyma

Answer 2

Have a look at the answer from @Spencer - it is the correct one. Here are some minor modifications for 2021 PEPs conventions and a modulo, to only update output every x MB (or KB, or GB).

import logging, enum
from paramiko import SSHClient

class SIZE_UNIT(enum.Enum):
   BYTES = 1
   KB = 2
   MB = 3
   GB = 4

def convert_unit(size_in_bytes: int, unit: SIZE_UNIT):
    """Convert the size from bytes to 
    other units like KB, MB or GB
    """
    if unit == SIZE_UNIT.KB:
        return size_in_bytes/1024
    elif unit == SIZE_UNIT.MB:
        return size_in_bytes/(1024*1024)
    elif unit == SIZE_UNIT.GB:
        return size_in_bytes/(1024*1024*1024)
    else:
        return size_in_bytes
        
def progress(transferred: int, tobe_transferred: int):
    """Return progress every 50 MB"""
    if convert_unit(transferred, SIZE_UNIT.MB) % 50 != 0:
        return
    logging.getLogger().info(
        f"Transferred: "
        f"{convert_unit(transferred, SIZE_UNIT.GB):.2f}GB \t"
        f"out of: {convert_unit(tobe_transferred, SIZE_UNIT.GB):.2f}"
        f"GB")

client = SSHClient()
client.connect(...)
sftp = client.open_sftp()
sftp.get(remotepath, localpath, callback=progress)

Jenellejenesia

Good one for logging current status of STFP download. – Kolyma