Archive all files from one SFTP folder to another in Python

Question

I was able to successfully upload the file from S3 to SFTP location using the below syntax as given by @Martin Prikryl Transfer file from AWS S3 to SFTP using Boto 3.

with sftp.open('/sftp/path/filename', 'wb') as f:
    s3.download_fileobj('mybucket', 'mykey', f)

I have a requirement to archive the previous file into the archive folder from the current folder before uploading the current dated file from S3 to SFTP

I am trying to achieve using the wildcard, because sometimes, when running on Monday, you won't be able to find the file for Sunday and you have the previous file which is Friday's file. So I want to achieve any of the previous file irrespective of the date.

Example

I have folder as below and filename_20200623.csv needs to be moved to ARCHIVE folder and the new file filename_20200625.csv will be uploaded.

MKT
  ABC
    ARCHIVE
    filename_20200623.csv

Expected

MKT
  ABC
    ARCHIVE
      filename_20200623.csv
    filename_20200625.csv

Deponent

Answer 1

Use Connection.listdir_attr to retrieve list of all files in the directory, filter it to those you are interested in, and then move them one-by-one using Connection.rename:

remote_path = "/remote/path"
archive_path = "/archive/path"
for f in sftp.listdir_attr(remote_path):
    if (not stat.S_ISDIR(f.st_mode)) and f.filename.startswith('prefix'):
        remote_file_path = remote_path + "/" + f.filename
        archive_file_path = archive_path + "/" + f.filename
        print("Archiving %s to %s" % (remote_file_path, archive_file_path))
        sftp.rename(remote_file_path, archive_file_path)

---

For future readers, who use Paramiko, the code will be identical, except of course that sftp will refer to Paramiko SFTPClient class, instead of pysftp Connection class. As Paramiko SFTPClient.listdir_attr and SFTPClient.rename methods behave identically to those of pysftp.

Eyde