What is limiting the # of simultaneous connections my ASP.NET application can make to a web service?

Question

I have an ASP.NET 4.0 application running atop IIS 7.5 on a 64-bit Windows Server 2008 R2 Enterprise machine with gobs of RAM, CPU, disk, etc.

With every web request, the ASP.NET application makes a connection to a backend web service (via raw sockets), which is running on the same machine.

Problem: There appears to be something limiting the # of simultaneous connections to the backend web service. Suspiciously, the number of concurrent connections is topping out at 16.

I found this key article from Microsoft explaining how to tweak IIS' settings to accomodate ASP.NET apps that make lots of web service requests: http://support.microsoft.com/?id=821268#tocHeadRef

I followed the article's recommendatinos, but still no luck. The setting that is particularly interesting is the maxconnection setting, which I even bumped to 999.

Any ideas what else could be throttling connections?

Note: When I cut IIS out of the mix and have the clients connect directly to the backend web service, it will happily open as many connections as I need, so I'm positive the backend is not the bottleneck. It must be something in IIS/ASP.NET-land.

Here's the relevant section of the machine.config which I'm sure is being read by the application (verified with appcmd.exe):

<system.web>
    <processModel autoConfig="false" maxWorkerThreads="100" maxIoThreads="100" minWorkerThreads="50" />
    <httpRuntime minFreeThreads="176" minLocalRequestFreeThreads="152"/>

    <httpHandlers />

    <membership>
        <providers>
            <add name="AspNetSqlMembershipProvider"
                type="System.Web.Security.SqlMembershipProvider, System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
                connectionStringName="LocalSqlServer"
                enablePasswordRetrieval="false"
                enablePasswordReset="true"
                requiresQuestionAndAnswer="true"
                applicationName="/"
                requiresUniqueEmail="false"
                passwordFormat="Hashed"
                maxInvalidPasswordAttempts="5"
                minRequiredPasswordLength="7"
                minRequiredNonalphanumericCharacters="1"
                passwordAttemptWindow="10"
                passwordStrengthRegularExpression="" />
        </providers>
    </membership>

    <profile>
        <providers>
            <add name="AspNetSqlProfileProvider" connectionStringName="LocalSqlServer" applicationName="/"
                type="System.Web.Profile.SqlProfileProvider, System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
        </providers>
    </profile>

    <roleManager>
        <providers>
            <add name="AspNetSqlRoleProvider" connectionStringName="LocalSqlServer" applicationName="/"
                type="System.Web.Security.SqlRoleProvider, System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
            <add name="AspNetWindowsTokenRoleProvider" applicationName="/"
                type="System.Web.Security.WindowsTokenRoleProvider, System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
        </providers>
    </roleManager>
</system.web>
<system.net>
    <connectionManagement>
        <add address="*" maxconnection="999"/>
    </connectionManagement>
</system.net>

Annihilator

@DanB has a good point here - how are you measuring the number of concurrent connections? – Vuillard

@JeremyMcGee I'm measuring the # of concurrent connections by running TCPView on the server to see how many backend connections are made by the IIS worker processes. – Annihilator

Are the Web clients running on independent machines or the same machine? (Checking that there's no client-side throttling going on.) – Vuillard

@JeremyMcGee Separate machines. 1 client-server connection per machine. Also, we know there isn't any throttling somewhere on the network because when we hit the backend directly (which happens over HTTP as well) we don't run into the bottleneck. – Annihilator

We had this same issue with a console app that called a webservice and needed to do lots of simultaneous calls. We discovered the console app was throttling itself to 2 connections. Setting System.Net.ServicePointManager.DefaultConnectionLimit = 1000; cleared the issue, but now we're using the same code on a website and I'm wondering how this setting relates to the solutions talked about here. Which one takes priority? – Houphouetboigny

Rob, did you ever find a definitive solution to this? – Tablecloth

Neither solution using maxconnection="65535" ? In ASP.NET 4.5 new configuration sections maybe ? – Transposal

Answer 1

Most of the answers provided here address the number of incoming requests to your backend webservice, not the number of outgoing requests you can make from your ASP.net application to your backend service.

It's not your backend webservice that is throttling your request rate here, it is the number of open connections your calling application is willing to establish to the same endpoint (same URL).

You can remove this limitation by adding the following configuration section to your machine.config file:

<configuration>
  <system.net>
    <connectionManagement>
      <add address="*" maxconnection="65535"/>
    </connectionManagement>
  </system.net>
</configuration>

You could of course pick a more reasonable number if you'd like such as 50 or 100 concurrent connections. But the above will open it right up to max. You can also specify a specific address for the open limit rule above rather than the '*' which indicates all addresses.

MSDN Documentation for System.Net.connectionManagement

Another Great Resource for understanding ConnectManagement in .NET

Hope this solves your problem!

EDIT: Oops, I do see you have the connection management mentioned in your code above. I will leave my above info as it is relevant for future enquirers with the same problem. However, please note there are currently 4 different machine.config files on most up to date servers!

There is .NET Framework v2 running under both 32-bit and 64-bit as well as .NET Framework v4 also running under both 32-bit and 64-bit. Depending on your chosen settings for your application pool you could be using any one of these 4 different machine.config files! Please check all 4 machine.config files typically located here:

• C:\Windows\Microsoft.NET\Framework\v2.0.50727\CONFIG
• C:\Windows\Microsoft.NET\Framework64\v2.0.50727\CONFIG
• C:\Windows\Microsoft.NET\Framework\v4.0.30319\Config
• C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Config

Kordofanian

Yup, I covered that base. Thanks anyway, @BenSwayne! I verified that I modified the correct machine.config (64-bit 4.0). – Annihilator

@RobSobers: In this case I would be a little suspicious of the implementation code. Perhaps you are running out of threads or something? Can you throw your TcpClient webservice call code into a console app and see if you can achieve a better request rate? This will prove whether its an IIS specific configuration or a broader .NET configuration or your code. – Kordofanian

Does this line go in the client machine? – Marabout

thank you , your settings combine with processModel twekaing from codeproject.com/Articles/133738/… fixed "ISAPI ‘C:\windows\Microsoft.Net\Framework\v2.0.050727\aspnet_isapi.dll’ reported itself as unhealthy for the following reason: ‘Deadlock detected" problem – Sabina

@Kordofanian does the same concept applies for SMTP connections as well? I am designing a bulk email sending application so will this ConnectionManagement property will be useful in sending bulk mails as well (using multi threading for mail.send function)? – Zipah

Answer 2

I realize the question might be rather old, but you say the backend is running on the same server. That means on a different port, probably other than the default port 80.

I've read that when you use the "connectionManagement" configuration element, you need to specify the port number if it differs from the default 80.

LINK: maxConnection setting may not work even autoConfig = false in ASP.NET

Secondly, if you choose to use the default configuration (address="*") extended with your own backend specific value, you might consider putting the specific value first! Otherwise, if a request is made, the * matches first and the default of 2 connections is taken. Just like when you use the section in web.config.

LINK: <remove> Element for connectionManagement (Network Settings)

Hope it helps someone.

Nourishing

Answer 3

Might it be possible that you're using a WCF-based web service reference? By default, the ServiceThrottlingBehavior.MaxConcurrentCalls is 16.

You could try updating your service reference behavior's <serviceThrottling> element

<serviceThrottling
    maxConcurrentCalls="999" 
    maxConcurrentSessions="999" 
    maxConcurrentInstances="999" />

(Not that I'd recommend the settings above.) See MSDN for more information how to configure an appropriate <behavior> element.

Boff

I wish we were, but we're not. The service being consumed is running in Apache/Python/mod_wsgi on another machine, and, as Rob mentioned, it's clearly not the issue. – Woodsy

Service References are on the client. How is your client consuming the Apache service? – Choate

Like John said: the throttling is set on the client consuming the web service. Perhaps the phrase "your service behavior" is a bit misleading. Does it make more sense when phrased as "your service reference behavior"? – Boff

@john It's consumed via a TcpClient (the service vends custom binary blobs, not WCF/SOAP or similar). Those configuration options seem to be purely impacting you if you use ServiceHost, not TcpClient; am I missing something? – Woodsy

Why not use "Add Service Reference"? – Choate

Answer 4

Have you tried to set the value of the static DefaultConnectionLimit property programmatically?

Here is a good source of information about that true headache... ASP.NET Thread Usage on IIS 7.5, IIS 7.0, and IIS 6.0, with updates for framework 4.0.

Feverous

I imagine this should not matter, but I'm going to try it anyway. – Annihilator

This is how we fixed it a few years back. We had concurrency issues and this seemed to clear it up. Net.ServicePointManager.DefaultConnectionLimit = 1000 is what we used. You must set it BEFORE you create connection classes. – Duppy

Answer 5

See the "Threading" section of this page: http://msdn.microsoft.com/en-us/library/ff647786.aspx, in conjunction with the "Connections" section.

Have you tried upping the maxconnection attribute of your processModel setting?

Enochenol

Yes, I have. But that article you cited points out something interesting that ever other article on the topic neglected to mention: that the maxconnection attribute does not apply to local web service calls. In this particular case the web service is local, but we have that same problem in another environment where the service is not local. – Annihilator

@RobSobers - I think that the above link is worthy of a second look. Check the part about minLocalRequestFreeThreads - This worker process uses this setting to queue up requests from localhost (where a Web application calls a Web service on the same server) if the number of available threads in the thread pool falls below this number. This setting is similar to minFreeThreads, but it only applies to requests that use localhost. – Chestonchest

@Chestonchest Yup, I set minLocalRequestFreeThreads as well (see my machine.config in the question. – Annihilator

Answer 6

If it is not defined in the web service or application or server (apache or IIS) that is hosting the web service consumable then you could create infinite connections until failure

Buccaneer

Answer 7

while doing performance testing, the measure i go by is RPS, that is how many requests per second can the server serve within acceptable latency.

theoretically one server can only run as many requests concurrently as number of cores on it..

It doesn't look like the problem is ASP.net's threading model, since it can potentially serve thousands of rps. It seems like the problem might be your application. Are you using any synchronization primitives ?

also whats the latency on your web services, are they very quick to respond (within microseconds), if not then you might want to consider asynchronous calls, so you dont end up blocking

If this doesnt yeild something, then you might want to profile your code using visual studio or redgate profiler

Meany