Prevent SFTP/SSH session timeout with paramiko

Question

I'm using paramiko to connect to an SFTP server on which I have to download and process some files.

The server has a timeout set to 5 minutes, but some days it happens that the processing of the files can take longer than the timeout. So, when I want to change the working directory on the server to process some other files sftp.chdir(target_dir)) I get an exception that the connection has timed out:

File buildbdist.win32eggparamikosftp://ftp.py, line 138, in _write_all raise EOFError()

To counter this I thought that activating the keep alive would be the best option so I used the "set_keepalive" on the transport to set it to 30 seconds:

ssh = paramiko.SSHClient()
ssh.set_missing_hostkey_policy(paramiko.AutoAddPolicy())
ssh.connect(ssh_server, port=ssh_port, username=ssh_user, password=password)
transport = ssh.get_transport()
transport.set_keepalive(30)
sftp = transport.open_sftp_client()

But nothing changes at all. The change has absolutely no effect. I don't know if I'm misunderstanding the concept of set_keepalive here or maybe the server (on which I have no access) ignores the keep alive packets.

Isn't this the right way to counter this problem or should I try a different approach? I don't like the idea of "manually" sending some ls command to the server to keep the session alive.

Alimentation

Keepalive packets are ignored my the server. They are only for keeping network connections from timing out. – Hellenistic

Answer 1

If the server is timing you out for inactivity, there's not much you can do from the client-side (other than perhaps send a simple command every now and again to keep your session from timing out).

Have you considered breaking apart your download and processing steps, so that you can download everything you need to start with, then process it either asynchronously, or after all downloads have completed?

Radiotelephone

ok, I see. I was also thinking about a redesign of the process so that the processing/download is async but that would be a quite bit of work for which I currently I have no time. Seems I'm unlucky that the solution with "keep alive" will not work. Another attempt I could try might by to make a reconnect to the server when I notice that the session has had a timeout. – Alimentation

Yes, reconnection would be a good solution. You could reconnect upon socket.error and EOFError, and that should suffice. – Radiotelephone

Also how to catch a disconnect? – Lagrange

Why is "sending a simple command now and then" doing nothing? How can I do that cleanly with paramiko? Why is Keepalive not just that? – Lagrange

Answer 2

This solution worked for me - starting a daemon thread that keeps the connection alive. Something like this:

from threading import Thread, Event
import time

# Function to keep the SFTP connection alive
def keep_sftp_alive(transport,stop_event, interval=60):
    while not stop_event.is_set():
        time.sleep(interval)
        transport.send_ignore()
    
stop_event= Event()
keep_alive_thread = Thread(target=keep_sftp_alive, args=(transport,stop_event,))
keep_alive_thread.daemon = True
keep_alive_thread.start()

<your main code here>

stop_event.set()

The daemon thread will only continue working if the main Python code is still running.

Nottinghamshire