Copy file from remote server using SFTP straight to Azure blob storage

Question

I've got a remote server with a bunch of static files (outside of Azure). I've created a worker role in Azure and I want to use the worker role to transfer these files using SFTP from my remote server straight to my blob storage account (without copying locally to the worker role). Is there an established workflow/best practices on how to do this?

The closest thing I was able to find was this question: Copy file from URL to Azure BLOB

However to use StartCopyFromBlob I would need to have a publicly accessible URL which is not the case.

Also some of these files may be >100mb or >500mb, should that raise any problems?

Thanks.

Streeto

See also: https://mcmap.net/q/536805/-windows-azure-and-sftp – Exultant

See also: #52841731 – Exultant

See also: #35050739 – Exultant

Answer 1

You might want to tackle this the other way around, by setting up an FTP server on Azure which save the files directly to Azure storage.

A good explanation on how to do this by use of the Preview Files can be found on http://fabriccontroller.net/blog/posts/deploying-a-load-balanced-high-available-ftp-server-with-azure-files/

Valois

Answer 2

The simplest approach would be to install the Azure CLI directly on the remote server. You can then use the Azure CLI to transfer these files directly to Azure Blob Storage. No need for SFTP (Azure Blob Storage doesn't expose an SFTP interface) or worker roles (your remote server isn't serving files over a web interface).

If this is not an option, the other approach would be to do what @Mark Volders suggested, and provision an SFTP server on Azure. You can then push files from the remote server to the SFTP server. The SFTP server would then push the file to Azure Blob Storage using the Azure CLI, and delete the local file upon success.

For the SFTP server, one hurdle is making sure files are copied to Azure Blob Storage as soon as the SFTP client is done transferring a file. A common approach is to use Incron, which is a service that listens for file events (in this case, the IN_CLOSE_WRITE event). There's a product SFTP Gateway on the Azure Marketplace that does all this (disclosure: I'm one of the developers for this product) so you don't need to spend time implementing this from scratch.

Also, file sizes of >500mb should not be a problem for either the Azure CLI or SFTP.

Plumbo

Answer 3

You can use SFTP hosted on a docker image hosted on ACI which pushes the files to the Azure Blob Storage container and later deletes them from its local storage after it's uploaded successfully onto the blob storage.

It's also possible to mount Azure Blob Storage using Blobfuse, but that requires some extra privileges for the docker image, which ACI doesn't provide.

I've written an article to tackle the same problem here: https://aws.plainenglish.io/sftp-as-caas-on-cloud-1171080aa5df

Dollarbird