SSH.Net Async file download

Question

I am trying to download files asynchronously from an SFTP-server using SSH.NET. If I do it synchronously, it works fine but when I do it async, I get empty files. This is my code:

var port = 22;
string host = "localhost";
string username = "user";
string password = "password";
string localPath = @"C:\temp";

using (var client = new SftpClient(host, port, username, password))
{
    client.Connect();
    var files = client.ListDirectory("");

    var tasks = new List<Task>();

    foreach (var file in files)
    {                        
        using (var saveFile = File.OpenWrite(localPath + "\\" + file.Name))
        {
            //sftp.DownloadFile(file.FullName,saveFile); <-- This works fine
            tasks.Add(Task.Factory.FromAsync(client.BeginDownloadFile(file.FullName, saveFile), client.EndDownloadFile));
        }                        
    }

    await Task.WhenAll(tasks);
    client.Disconnect();

}

Sclaff

Answer 1

Because saveFile is declared in a using block, it is closed right after you start the task, so the download can't complete. Actually, I'm surprised you're not getting an exception.

You could extract the code to download to a separate method like this:

var port = 22;
string host = "localhost";
string username = "user";
string password = "password";
string localPath = @"C:\temp";

using (var client = new SftpClient(host, port, username, password))
{
    client.Connect();
    var files = client.ListDirectory("");

    var tasks = new List<Task>();

    foreach (var file in files)
    {                        
        tasks.Add(DownloadFileAsync(file.FullName, localPath + "\\" + file.Name));
    }

    await Task.WhenAll(tasks);
    client.Disconnect();

}

...

async Task DownloadFileAsync(string source, string destination)
{
    using (var saveFile = File.OpenWrite(destination))
    {
        var task = Task.Factory.FromAsync(client.BeginDownloadFile(source, saveFile), client.EndDownloadFile);
        await task;
    }
}

This way, the file isn't closed before you finish downloading the file.

---

Looking at the SSH.NET source code, it looks like the async version of DownloadFile isn't using "real" async IO (using IO completion port), but instead just executes the download in a new thread. So there's no real advantage in using BeginDownloadFile/EndDownloadFile; you might as well use DownloadFile in a thread that you create yourself:

Task DownloadFileAsync(string source, string destination)
{
    return Task.Run(() =>
    {
        using (var saveFile = File.OpenWrite(destination))
        {
            client.DownloadFile(source, saveFile);
        }
    }
}

Yorkshire

Thank you for the answer, however, I still get the same empty files when I try this. No exception either. – Sclaff

@spersson, I updated my answer. Looks like there is no advantage in using BeginDownloadFile, so you might as well use the synchronous version. – Yorkshire

Thanks for taking the time, I guess I'll use the synchronous version then :) – Sclaff

Though it is a bummer that they don't use IO completion ports in their implementation, the advantage to using the Begin* async overloads over creating your own thread is that some of their IAsyncResult implementations expose a cancellation mechanism, should you need to halt transfers midway through. I wrapped BeginUploadFile in an extension method to make things feel more modern. Check it out: gist.github.com/ronnieoverby/438034b19531e6272f98 – Tu

Answer 2

They finally updated the methods a while back but didn't add explicit DownloadFileAsync as was originally requested. You can see the discussion here:

https://github.com/sshnet/SSH.NET/pull/819

Where the solution provided is:

/// <summary>
/// Add functionality to <see cref="SftpClient"/>
/// </summary>
public static class SftpClientExtensions
{
    private const int BufferSize = 81920;

    public static async Task DownloadFileAsync(this SftpClient sftpClient, string path, Stream output, CancellationToken cancellationToken)
    {
        await using Stream remoteStream = await sftpClient.OpenAsync(path, FileMode.Open, FileAccess.Read, cancellationToken).ConfigureAwait(false);
        await remoteStream.CopyToAsync(output, BufferSize, cancellationToken).ConfigureAwait(false);
    }

    public static async Task UploadFileAsync(this SftpClient sftpClient, Stream input, string path, FileMode createMode, CancellationToken cancellationToken)
    {
        await using Stream remoteStream = await sftpClient.OpenAsync(path, createMode, FileAccess.Write, cancellationToken).ConfigureAwait(false);
        await input.CopyToAsync(remoteStream, BufferSize, cancellationToken).ConfigureAwait(false);
    }
}

Nic

Should be the right answer tbh (at least the most up to date one). Hope every other people could accept this one for other people struggling doing an async download with this package. – Goldner

Answer 3

I found that this solution (mentioned in another answer) did work, but it was extremely slow. I've tried switching the BufferSize a bunch of times but the performance was nothing compared to the default DownloadFile method.

I've found another solution which allows the download to be run asynchronously without the bottleneck:

public static class SftpClientExtensions
{
    public static async Task DownloadFileAsync(this SftpClient client, string source, string destination, CancellationToken cancellationToken)
    {
        TaskCompletionSource<bool> tcs = new();

        using (FileStream saveFile = File.OpenWrite(destination))
        // cancel the tcs when the token gets cancelled
        using (cancellationToken.Register(() => tcs.TrySetCanceled()))
        {
            // begin the asynchronous operation
            client.BeginDownloadFile(source, saveFile, result =>
            {
                try
                {
                    // Try to complete the operation
                    client.EndDownloadFile(result);

                    // indicate success
                    tcs.TrySetResult(true);
                }
                catch (OperationCanceledException)
                { 
                    // properly handle cancellation
                    tcs.TrySetCanceled();
                }
                catch (Exception ex)
                {
                    // handle any other exceptions
                    tcs.TrySetException(ex);
                }
            }, null);

            // await the task to complete or be cancelled
            await tcs.Task;

            // since we catch the cancellation exepction in the task, we need to throw it again
            cancellationToken.ThrowIfCancellationRequested();
        }
    }
}

Curettage