What's the "right way" to use HttpClient synchronously?

Question

I used quote marks around "right way" because I'm already well aware that the right way to use an asynchronous API is to simply let the asynchronous behavior propagate throughout the entire call chain. That's not an option here.

I'm dealing with a very large and complicated system designed specifically to do batch processing synchronously in a loop.

The reason why suddenly I'm using HttpClient is because prior to now all data for the batch processing was gathered from a SQL database, and now we're adding a Web API call to the mix.

Yes, we're calling a Web API in a synchronously executing loop. I know. Rewriting the whole thing to be async just isn't an option. This is actually what we want to do. (We're minimizing the number of API calls as much as possible)

I actually did try to propagate the async behavior up the call chain, but then I found myself 50 files deep in changes, still with hundreds of compiler errors to resolve, and lost all hope. I am defeated.

So then, back to the question, given Microsoft's recommendation to never use WebRequest for new development and to instead use HttpClient, which offers only an asynchronous API, what am I to do?

Here is some pseudo-code of what I'm doing...

foreach (var thingToProcess in thingsToProcess)
{
    thingToProcess.ProcessStuff(); // This makes an API call
}

How do I implement ProcessStuff()?

My first implementation looked like this

public void ProcessStuff()
{
    var apiResponse = myHttpClient // this is an instance of HttpClient
        .GetAsync(someUrl)
        .Result;

    // do some stuff with the apiResponse
}

I was told however, that calling .Result in this manner can result in deadlocks when it's called from something like ASP.NET due to the synchronization context.

Guess what, this batch process will be kicked off from an ASP.NET controller. Yes, again, I know, this is silly. When it runs from ASP.NET it's only "batch processing" one item instead of the whole batch, but I digress, it still gets called from ASP.NET and thus I'm concerned about deadlocks.

So what's the "right way" to handle this?

Gokey

Possible duplicate of: #22628587 – Stealthy

Possible duplicate of Calling async method synchronously – Memorialize

Answer 1

For anyone coming across this now, .NET 5.0 has added a synchronous Send method to HttpClient. https://github.com/dotnet/runtime/pull/34948

You can therefore use this instead of SendAsync. For example

public string GetValue()
{
    var client = new HttpClient();
            
    var webRequest = new HttpRequestMessage(HttpMethod.Post, "http://your-api.com")
    {
        Content = new StringContent("{ 'some': 'value' }", Encoding.UTF8, "application/json")
    };

    var response = client.Send(webRequest);

    using var reader = new StreamReader(response.Content.ReadAsStream());
            
    return reader.ReadToEnd();
}

This code is just a simplified example, it's not production ready.

Catawba

I cry b/c our enterprise software still uses netfx 4.7 – Combinative

@Combinative You're not alone) – Oligopsony

The fact that this hasn't been back ported to the .NET Framework in some way, where it would actually be incredibly useful, is very very frustrating. – Hypnogenesis

4.7? Luxury.... – Dardan

.Net CORE / Standard 7.0 lacks this... – Fourpenny

elegantly beautiful as .NET always is. A httpClient.Send(url) would have been too cumbersome for sure – Unwelcome

@Combinative Have a look at my answer, this may help you out. – Ivonne

@Oligopsony Have a look at my answer, this may help you out. (sorry for spamming - only one user notification per comment allowed) – Ivonne

@BradfordDillon Have a look at my answer, this may help you out. (sorry for spamming - only one user notification per comment allowed) – Ivonne

@Dardan Have a look at my answer, this may help you out. (sorry for spamming - only one user notification per comment allowed) – Ivonne

Answer 2

Try the following:

var task = Task.Run(() => myHttpClient.GetAsync(someUrl)); 
task.Wait();
var response = task.Result;

Use it only when you cannot use an async method.

This method is completely deadlock free as mentioned on the MSDN blog: ASP.Net–Do not use Task .Result in main context.

Memorialize

Thank you for finally providing a definitive answer to this in the form of the MSDN post. I actually Googled this before posting and found many similar questions on StackOverflow, but none of them provided a source from MS that explicitly stated that running the task on a thread, waiting, and fetching the result after waiting is safe. Many other SO threads on this question actually have the wrong answer as the accepted answer, but it looks very similar to the right answer. That is, they skip the call to .Wait() – Gokey

This doesn't work. Deadlock on task.Wait(); – Berns

2021 update: See the other answer from @alexs which I have now marked as the accepted answer. There's a new synchronous method added to the HttpClient API. – Gokey

@Gokey I think this is still the better answer for the specific question you asked, which is likely to draw people for whom "use .NET 5" is not a reasonable answer to get one little piece of code working. For them, the fundamental premise of the question is still very relevant: there is no synchronous API, but the alternatives are deprecated. I doubt many people using .NET 5 would even have a reason to ask this question. Just a thought. – Drover

Might be obvious for some, but wasn't for me and wanted to note for others just in case - add using System.Threading.Tasks; to your code. I chose the wrong suggested reference and wanted to save someone else the confusion. – Considered

It's certainly not "deadlock free" - for anyone struggling with most of the misinformation on this see the answer from alexs – Fisticuffs

myHttpClient.GetAsync(someUrl).Result should literally be all you need to make it synchronous. – Brut

If you are using a version < .Net 5.0 and using the synchronous Send method is not an option then using .Wait(timeOutMilliSeconds); may help avoiding deadlocks. – Elated

Answer 3

You could also look at using Nito.AsyncEx, which is a nuget package. I've heard of issues with using Task.Run() and this this addresses that. Here's a link to the api docs: http://dotnetapis.com/pkg/Nito.AsyncEx/4.0.1/net45/doc/Nito.AsyncEx.AsyncContext

And here's an example for using an async method in a console app: https://blog.stephencleary.com/2012/02/async-console-programs.html

Stealthy

The only solution that worked for me (not using .NET 5,6, etc). The rest of the answers just end up on the main thread and if the http call hangs, the main thread hangs. – Roofing

Answer 4

In the meantime, .NET 6.0 has received sync support on HttpClient (see GitHub thread on the topic).

However, users of the old .NET Framework are left out. I therefore implemented a library which does add sync support to the legacy HttpClient by implementing a custom HttpClientHandler and doing some trickery to make the HttpContent calls (mostly) execute synchronously. This should prevent deadlocks and threadpool starvation issues in many cases - your mileage may vary.

// Initialize the HttpClient with the custom handler
var client = new HttpClient(new HttpClientSyncHandler());

// Invoke sync HTTP call
using var request = new HttpRequestMessage(HttpMethod.Get, "https://1.1.1.1/");
using var response = client.Send(request);
// use the response here
// response.Content.ReadAsStream()
// response.Content.ReadAsByte[]()
// response.Content.ReadAsString()

Source: https://github.com/avonwyss/bsn.HttpClientSync - it's also available as NuGet package.

Ivonne

Answer 5

RestSharp has an AsyncHelper that allows one to make sync calls to async methods (RestSharp in turn borrowed that class from Rebus).

I have used that class in the past (I literally just Copy&Pasted it) to make sync calls to async method and it works like charm. In case you are wondering why and how this works, there is a Blog-Post by Stephen Toub that explains how SynchronizationContext and ConfigureAwait(false) works.

To use it with the HttpClient you would do this:

AsyncHelpers.RunSync(async () => await httpClient.SendAsync(...));

If you intend to make a library/app that supports both .NET-Framework and .NET-Core you could further optimize that to this:

#if NETFRAMEWORK

//.NET-Framework does not support a sync Send
AsyncHelpers.RunSync(async () => await httpClient.SendAsync(...));

#elif NETCOREAPP

//.NET-Core does
httpClient.Send(...);

#else

//Default: Fallback to something that works on all targets.
AsyncHelpers.RunSync(async () => await httpClient.SendAsync(...));

#endif

For the sake of completeness here is AsyncHelper (again, no my implementation. I copied it from RestSharp and stripped the comments for brevity).

static class AsyncHelpers {

        public static void RunSync(Func<Task> task) {
            var currentContext = SynchronizationContext.Current;
            var customContext  = new CustomSynchronizationContext(task);

            try {
                SynchronizationContext.SetSynchronizationContext(customContext);
                customContext.Run();
            }
            finally {
                SynchronizationContext.SetSynchronizationContext(currentContext);
            }
        }
        
        public static T RunSync<T>(Func<Task<T>> task) {
            T result = default!;
            RunSync(async () => { result = await task(); });
            return result;
        }
        
        class CustomSynchronizationContext : SynchronizationContext {
            readonly ConcurrentQueue<Tuple<SendOrPostCallback, object?>> _items            = new();
            readonly AutoResetEvent                                      _workItemsWaiting = new(false);
            readonly Func<Task>                                          _task;
            ExceptionDispatchInfo?                                       _caughtException;
            bool                                                         _done;
            
            public CustomSynchronizationContext(Func<Task> task) =>
                _task = task ?? throw new ArgumentNullException(nameof(task), "Please remember to pass a Task to be executed");
            
            public override void Post(SendOrPostCallback function, object? state) {
                _items.Enqueue(Tuple.Create(function, state));
                _workItemsWaiting.Set();
            }
            
            public void Run() {
                async void PostCallback(object? _) {
                    try {
                        await _task().ConfigureAwait(false);
                    }
                    catch (Exception exception) {
                        _caughtException = ExceptionDispatchInfo.Capture(exception);
                        throw;
                    }
                    finally {
                        Post(_ => _done = true, null);
                    }
                }

                Post(PostCallback, null);

                while (!_done) {
                    if (_items.TryDequeue(out var task)) {
                        task.Item1(task.Item2);
                        if (_caughtException == null) {
                            continue;
                        }
                        _caughtException.Throw();
                    }
                    else {
                        _workItemsWaiting.WaitOne();
                    }
                }
            }
            
            public override void Send(SendOrPostCallback function, object? state) => throw new NotSupportedException("Cannot send to same thread");
            
            public override SynchronizationContext CreateCopy() => this;
        }
    }

Fairground