IIS ASP.NET WebApi Deadlock when requesting the same server

Question

We've been experiencing some deadlocks when working with interconnected ASP.NET WebApis on the same IIS server. We'd like to know if this is somehow an expected behavior, due to hosting all APIs on the same server and same Application Pool, since we have managed to avoid the issue by moving either WebApi to a different pool; or if there's something wrong with our code.

For production, we will probably host the APIs on different server or pools, but still we'd like to understand why this is happening. Our main concern is that, if it's our faulty code, the issue may be reproduced on a larger scale, even if the hosting setup is correct.

We have created a little solution to reproduce the deadlock, hosted in GitHub.

The reproduction steps are as follow:

1. WebClient executes multiple HTTP request in parallel WebApi1.
2. WebApi1 executes an HTTP request to WebApi2.
3. WebApi2 executes an HTTP request to WebApi3.
4. WebApi3 simply returns a string.

The expected behavior would be that all requests are eventually resolved.

The actual behavior is that, certain requests gets completed, while some others will fail, due to a TaskCancelledException which seems to be due to the requests timing out.

The only article that I was able to find that seems to mention the same issue is from 2014: "Do Not Send ServerXMLHTTP or WinHTTP Requests to the Same Server", I believe that this is the issue we are experiencing, how can we confirm this?

Context

We've been assigned the task to create a centralized authentication server for multiple internal APIs of the company we work at. We are using IdentityServer3 with reference tokens, so when some API requests a second API using reference tokens, the second API will request the authentication server for token validation which reproduces the issue.

I have added the IdentityServer tag, since this could be a common issue when doing multiple APIs communication and using reference tokens. Sample on GitHub.

Haddock

"how can we confirm this?" running each in a different pool and checking if the issue still exists maybe? – Closefisted

Hey @vtortola, we've already separated the WebApis into different pools, and the issue apparently disappears (10k parallel requests can't break it, while on the same pool 5 might). Still, we'd like to know why is the deadlock happening, and if it's an expected behavior with such a setup. – Haddock

I'd want to see some logs to find out more where it might be coming from. Is it possible that the deadlock at startup time when trying to load the metadata? Have you tried the DelayLoadMetadata flag? identityserver.github.io/Documentation/docsv2/consuming/… – Stamin

Hey @BrockAllen, nice to see you've joined to the party. We did read about the deadlock when loading metadata, and we actually experienced that issue ourselves and solved it by enabling the flag. I've come up with a project using IdentityServer3 which reproduces the exact issue we are experiencing. It's been done using the provided samples, github.com/leonardochaia/IdentityServerDeadlock – Haddock

Maybe you're hitting the limits of IIS? blogs.msdn.microsoft.com/benjaminperkins/2011/11/14/… – Wince

Hey @JohnKorsnes, we did increase the settings mentioned in the article and obtained the same results. I don't think we are hitting those limits. 5 request from the client to WebApi1, 5 requests from WebApi1 to WebApi2, 5 requests from WebApi2 to WebApi 3. There're only 15 requests, and this produces the deadlock. Besides that, we do not get a 503, WebApi's internal requests are being cancelled by timeout. – Haddock

The article about WinHTTP is for the unmanaged world. However, there is the same kind of default limit with .NET (this comes from the HTTP standard in fact) Can you try to set ServicePointManager.DefaultConnectionLimit to something like 1000 or whatever in your web servers startup code. msdn.microsoft.com/en-us/library/office/… – Ashleighashlen

Hey @SimonMourier, I've set the DefaultConnectionLimit to 2000, and performed the same tests again with the same results, the deadlock still occurs. – Haddock

I can't reproduce with your test solution. What are we supposed to observe? – Ashleighashlen

Hey @SimonMourier, nice to see you've downloaded the project. If you already met all the requirements in the WebClient project, you should only execute one of the Request X Times button. 10-50 parallel requests should be enough to produce the deadlock. We use the Chrome Developer Console for watching the requests results. – Haddock

I can run all the tests w/o any issue (takes ~5 sec for 2000 request for example) – Ashleighashlen

Hey @SimonMourier, that's weird. Are you running the APIs on the same Application Pool? Is the pool configured by default, specifically does that pool have a single Worker Process? We have tested the provided source code on different machines across the team, and we all had the same issue.. – Haddock

it's a pretty standard DefaultAppPool in the DefaultWebSite everywhere (Windows 10) – Ashleighashlen

Let us continue this discussion in chat. – Haddock

Answer 1

Just one observation: you are using HttpClient as a static member for every Controller and according to this HttpClient is not guaranteed to be thread-safe

Brush

Hey Dave, I've found this article from MSDN where they recommend creating a single HttpClient and reusing it, we've tried creating single instances wrapped in a using statement, with no changes. – Haddock