WCF Concurrent requests piling up on the server when using WSHttpBinding

Question

I have a WCF client/server app which is communicating over HTTP using the WSHttpBinding.

Server Setup: self-hosting, using the standard WCF ServiceHost. My actual service class is attributed as:

[ServiceBehavior(ConcurrencyMode = ConcurrencyMode.Multiple, 
 InstanceContextMode = InstanceContextMode.PerSession, 
 UseSynchronizationContext = false)]

Client Setup: using a visual-studio generated client proxy using synchronous service calls (proxy.call_server_method blocks until such time as the server has responded in full.)

Scenario: I have one particular method call which takes 20 seconds to execute on the server. The client calls this method in a separate thread, so it is not being held up, and the ConcurrencyMode.Multiple means WCF should execute it in a separate thread on the server as well.

This theory is supported by the fact that when I configure my app to use NetTcpBinding, everything works fine.

Problem:
If I configure the app to use WSHttpBinding, then this long method call causes the http requests to 'back up'. I have verified this behaviour both from inspecting my logs, and by debugging the HTTP requests using fiddler.

Example:

• Client initiates 20-second long request on a background thread
• Client initiates request B and C on foreground thread
• Requests B and C get sent to the server, which doesn't process them until it is done with the 20-second long request

But sometimes:

• Requests B and C do not get sent (they don't even appear in fiddler) until the 20-second request comes back (this is rare).
  • Note: setting <add address="*" maxconnection="100"/> in the client's app.config made this (appear to) stop happening.
• Request B gets sent and receives a response immediately, while request C is held back until the 20-second one completes (this is rare)

Here's a timeline from fiddler demonstrating the problem: (click for bigger version)

As you can see, the requests are all getting backed up at the server. Once the 20-second request completes, the responses all come flooding through, but note that there are some requests which aren't getting held up...

So, Questions:

• What the heck is going on here? Why does it work fine using NetTcpBinding and not work using WSHttpBinding?
• Why the inconsistent behaviour?
• What can I do to fix it?

Notes:

• It's not locking on the server. I've set breakpoints and used !syncblk and it consistently reports no locks are being held.
• It's not my threading (NetTcpBinding shouldn't work otherwise)
• I have <serviceThrottling maxConcurrentCalls="1000" maxConcurrentInstances="1000" maxConcurrentSessions="1000" /> set in the server's app.config
• The 20-second call is just waiting on a timer, it's not thrashing the CPU or disk or network
• I'd prefer a solution that didn't involve re-architecting the application to use asynchronous calls... it's a big bunch of legacy code and I really don't want to be messing with stuff I don't understand.

Uird

+1 Excellent layout of the question. – Sarina

Did you ever solve this issue? If so, how did you solve it? – Defective

Added a self-answer describing our final "solution" – Uird

Answer 1

[Self-answer to show other users what our eventual solution was]

In the end, I never managed to solve this.
Our final solution was to switch our app away from WSHttpBinding and onto NetTcpBinding in production - we had been planning on doing this eventually anyway for performance reasons.

This is rather unfortunate though, as it leaves a black mark on WSHttpBinding which may or may not be warranted. If anyone does ever come up with a solution which does not involve ditching WSHttpBinding, I'd love to know about it

Uird

I think you shouldn't self-answer as actually we have no answer. Have you try to post bug to connect.microsoft.com ? – Acquiescence

Now that I've learnt a bit more about how things work, I think it may have been related to overloading the ThreadPool. If you queue lots of long-running operations to the threadpool, you get behaviour that looks almost exactly like this – Uird

I'm having exactly the same issue with WSHttpBinding and I don't think it is due to an overloaded thread pool. I used NetTcpBinding and it suddenly works perfectly. I can get what you describe simply by making my first call to the service an sleeping it whilst trying to display another call in parallel from another server. Nothing else can contact the service irrespective of what my service behaviour and service throttling has been set to. – Koblenz

Answer 2

There is some throttle outside of WCF (a .Net or Windows thing) that defaults to only letting a maximum of two simultaneous outbound HTTP connections. Unfortunately I can't remember for the life of me the name of the thing (and what you'd put in app.config or your app to override it). Given that you're not seeing the requests leave the client, and that it's only HTTP, I think you're hitting "that thing". I'll keep looking for its name.

Update: Found it - try this on the client (but change the '2' to a bigger number):

<configuration>
  <system.net>
    <connectionManagement>
      <add address = "*" maxconnection = "2" />
    </connectionManagement>
  </system.net>
</configuration>

Keifer

Unfortunately this didn't solve it... It changed the behaviour somewhat, but now all the requests seem to back up at the server – Uird

Updated my question with a timeline graph from fiddler which should hopefully help – Uird

I want to deploy wcf manually to iis, and consume using some other client, will I be able to do it like this? – Trenton

Answer 3

We saw exactly the same symptoms with a JSON service hosted in IIS/ASP.NET.

The root cause ended up being ASP.NET synchronizing the requests - not WCF. We had to disable session state (at the application level) to get concurrent WCF methods.

Web.config: <system.web> <sessionState mode="Off" /> </system.web>

Please note that our service uses webHttpBinding, not wsHttpBinding. So, I'm not sure if this also solves Orion's problem.

Betelgeuse

Answer 4

I think you hit the protocol limit and to work around it you need to modify the standard settings on the client machine:

http://support.microsoft.com/kb/183110

http://support.microsoft.com/kb/282402

I guess WSHttpBinding uses WinINET settings when issuing the requests.

Ostap

This seems likely, but I have long since lost the original code. Thanks though! – Uird

Answer 5

If you change to BasicHttpBinding, does it work?

Is so, it sounds like this is your problem, Session throttling, something that bit me in the ass.

Hathcock

Answer 6

Consider using ConcurrencyMode.Multiple on per-call services to allow concurrent calls.

Santiago

Answer 7

I forget - could it be ordering? I think maybe RM over http preserves order but maybe Tcp sessions don't (unless you explicitly request it)? Is there an attribute on the Service Contract that described ordered/unordered sessions (I forget).

Keifer

There is <reliableSession> which has an ordered attribute... I have it set to <reliableSession ordered="false" enabled="true" /> – Uird

Answer 8

Not sure but sometimes the problem with the concurrent calls from silverlight application are related with browser connection management. For me the solution was to put that in our App.xaml.cs, Application_Startup method as descrive here: http://weblogs.asp.net/olakarlsson/simultaneously-calling-multiple-methods-on-a-wcf-service-from-silverlight

WebRequest.RegisterPrefix("http://", WebRequestCreator.ClientHttp);

Envisage