HttpWebResponse won't scale for concurrent outbound requests

Question

I have an ASP.NET 3.5 server application written in C#. It makes outbound requests to a REST API using HttpWebRequest and HttpWebResponse.

I have setup a test application to send these requests on separate threads (to vaguely mimic concurrency against the server).

Please note this is more of a Mono/Environment question than a code question; so please keep in mind that the code below is not verbatim; just a cut/paste of the functional bits.

Here is some pseudo-code:

// threaded client piece
int numThreads = 1;
ManualResetEvent doneEvent;

using (doneEvent = new ManualResetEvent(false))
        {

            for (int i = 0; i < numThreads; i++)
            {

                ThreadPool.QueueUserWorkItem(new WaitCallback(Test), random_url_to_same_host);

            }
            doneEvent.WaitOne();
        }

void Test(object some_url)
{
    // setup service point here just to show what config settings Im using
    ServicePoint lgsp = ServicePointManager.FindServicePoint(new Uri(some_url.ToString()));

        // set these to optimal for MONO and .NET
        lgsp.Expect100Continue = false;
        lgsp.ConnectionLimit = 100;
        lgsp.UseNagleAlgorithm = true;
        lgsp.MaxIdleTime = 100000;        

    _request = (HttpWebRequest)WebRequest.Create(some_url);


    using (HttpWebResponse _response = (HttpWebResponse)_request.GetResponse())
    {
      // do stuff
    } // releases the response object

    // close out threading stuff

    if (Interlocked.Decrement(ref numThreads) == 0)
    {
        doneEvent.Set();
    }
}

If I run the application on my local development machine (Windows 7) in the Visual Studio web server, I can up the numThreads and receive the same avg response time with minimal variation whether it's 1 "user" or 100.

Publishing and deploying the application to Apache2 on a Mono 2.10.2 environment, the response times scale almost linearly. (i.e, 1 thread = 300ms, 5 thread = 1500ms, 10 threads = 3000ms). This happens regardless of server endpoint (different hostname, different network, etc).

Using IPTRAF (and other network tools), it appears as though the application only opens 1 or 2 ports to route all connections through and the remaining responses have to wait.

We have built a similar PHP application and deployed in Mono with the same requests and the responses scale appropriately.

I have run through every single configuration setting I can think of for Mono and Apache and the ONLY setting that is different between the two environments (at least in code) is that sometimes the ServicePoint SupportsPipelining=false in Mono, while it is true from my machine.

It seems as though the ConnectionLimit (default of 2) is not being changed in Mono for some reason but I am setting it to a higher value both in code and the web.config for the specified host(s).

Either me and my team are overlooking something significant or this is some sort of bug in Mono.

Batrachian

Have you found a solution? – Overcast

Answer 1

I believe that you're hitting a bottleneck in the HttpWebRequest. The web requests each use a common service point infrastructure within the .NET framework. This appears to be intended to allow requests to the same host to be reused, but in my experience results in two bottlenecks.

First, the service points allow only two concurrent connections to a given host by default in order to be compliant to the HTTP specification. This can be overridden by setting the static property ServicePointManager.DefaultConnectionLimit to a higher value. See this MSDN page for more details. It looks as if you're already addressing this for the individual service point itself, but due to the concurrency locking scheme at the service point level, doing so may be contributing to the bottleneck.

Second, there appears to be an issue with lock granularity in the ServicePoint class itself. If you decompile and look at the source for the lock keyword, you'll find that it uses the instance itself to synchronize and does so in many places. With the service point instance being shared among web requests for a given host, in my experience this tends to bottleneck as more HttpWebRequests are opened and causes it to scale poorly. This second point is mostly personal observation and poking around the source, so take it with a grain of salt; I wouldn't consider it an authoritative source.

Unfortunately, I did not find a reasonable substitute at the time that I was working with it. Now that the ASP.NET Web API has been released, you may wish to give the HttpClient a look. Hope that helps.

Culler

Jesse, thanks for the info. Setting the ServicePointManager DefaultConnectionLimit still simply sets it per ServicePoint/endpoint so I dont know if that would bypass the locking issue you describe at the SP level. I would certainly expect ServicePoint to bottleneck as the threads/requests grow but not so dramatically. And it still seems like the requests go out no problem but the responses wait. I will have to look into HttpClient as an alternative. – Batrachian

Re: DefaultConnectionLimit - I don't claim that it will work miracles, but it does bypass a call to lock(this) in the setter for ConnectionLimit in ServicePoint itself. As for the scaling, my impression was the same. I still can't claim to truly understand why the bottleneck hits so hard. There are some synchronization constructs around getting the response and again if getting the streams. In your case, I would speculate the drop from local to non-local is due to the additional latency with an external server... but, again, that is pure speculation. – Culler

DefaultConnectionLimit didn't change anything (and removing the ConnectionLimit on the SP object reverted it to the default of 2). The diff from local to non-local is definitely Mono/environment related since the PHP script had no problems - same requests. That's what Im trying to get to the bottom of, under the assumption that it is in how Mono is handling these responses. – Batrachian

For the record, we are not (currently) using the Async methods for BeginGetResponse,etc and I think this may be the issue. As the "non-async" GetResponse still uses IOCP (I/O Completion Ports) which seem to vary by OS which would explain our environment performance differences. Using the Async methods seems to work on a cursory test. Reference: link – Batrachian

Interesting. I hadn't realized that the non-async methods still utilized I/O completion ports. As you've mentioned, that would need to be something that Mono accounts for which the standard framework does not. Unfortunately, that about runs me out of top-of-my-head ideas other than peeking at the Mono source. If I can carve out some free time, I may try just for giggles but I don't expect any ah-ha moments. Have you considered opening a ticket with the Mono development team for this? – Culler

I have not - I want to make sure Ive covered everything before bugging the dev team. For example, another snag in this equation seems to be SSL where our standard HTTP requests scale (via async) appropriately but HTTPS calls do not. Still trying to determine if that is a Mono issue or more of network issue on our end. If I can't isolate it to our environment, then I may peg the Mono group. Thanks for your assistance on this! – Batrachian

My pleasure. Sorry that I wasn't able to help nail down a definitive. Good luck. – Culler

I think the response of this guy is not taking in account that the question is about Mono, so downvoting – Bouncer

@Bouncer - You're entitled to your opinion, of course, but I respectfully disagree. The discussion centered around using the documented public interface of a class common to both Mono and the CLR to work around the issue. I'm not sure what you would have expected this to look like had I "taken into account that the question is about Mono." A down vote seems unwarranted in this situation, IMHO. – Culler

Answer 2

I know this is pretty old but I'm putting this here in case it might help somebody else who runs into this issue. We ran into the same problem with parallel outbound HTTPS requests. There are a few issues at play.

The first issue is that ServicePointManager.DefaultConnectionLimit did not change the connection limit as far as I can tell. Setting this to 50, creating a new connection, and then checking the connection limit on the service point for the new connection says 2. Setting it on that service point to 50 once appears to work and persist for all connections that will end up going through that service point.

The second issue we ran into was with threading. The current implementation of the mono thread pool appears to create at most 2 new threads per second. This is an eternity if you are doing many parallel requests that start at exactly the same time. To counteract this, we tried setting ThreadPool.SetMinThreads to a higher number. It appears that Mono only creates up to 1 new thread when you make this call, regardless of the delta between the current number of threads and the desired number. We were able to work around this by calling SetMinThreads in a loop until the thread pool had the desired number of idle threads.

I opened a bug about the latter issue because that's the one I'm most confident is not working as intended: https://bugzilla.xamarin.com/show_bug.cgi?id=7055

Stable

Can confirm ServicePointManager.DefaultConnectionLimit has no effect in Mono, limit is still 2 – Lambertson

Setting it in app.config or web.config works ... however is proving counter productive in Mono with anything other than 2 reducing throughput massively (?!). Also ServicePoint.Connections always reports 0 whereas Windows reports up to and just over the ConnectionLimit... – Lambertson

Answer 3

If @jake-moshenko is right about ServicePointManager.DefaultConnectionLimit not having any effect if changed in Mono, please file this as a bug in http://bugzilla.xamarin.com/.

However I would try some things before discarding this completely as a Mono issue:

1. Try using the SGen garbage collector instead of the old boehm one, by passing --gc=sgen as a flag to mono.
2. If the above doesn't help, upgrade to Mono 3.2 (which BTW defaults to SGEN GC too), because there has been a lot of fixes since you asked the question.
3. If the above doesn't help, build your own Mono (master branch), as this important pull request about threading has been merged recently.
4. If the above doesn't help, build your own Mono with this pull request added. If it fixes your problem, please add a "+1" to the pull request. It might be a fix for bug 7055.

Bouncer