NewRelic, async http handler and AcquireRequestState

Question

I have a problem with one async handler in distributed ASP.NET web app. First let me explain a use case:

• application uses IIS 8 on win 2012 machine with .NET Framework 4.5.2

• application has disabled Session and authentication modules via web.config like this

       <system.webServer>
         ....
         <modules>
              <remove name="WindowsAuthentication" />
              <remove name="Session" />
              <remove name="FormsAuthentication" />
          </modules>
       </system.webServer>
  

• application uses custom async web handler to serve the specific request

• application has very heavy traffic (about 50k requests per minute per server, async handler has about 10k requests per minute per server all tracked from NewRelic)
• application is distributed via multiple w3wp processes (2 w3wp processes) and multiple virtual servers (about 10 servers)
• application has high amount of connections

All normal (sync requests) are working fine but async request that does a little more work (that's why we use async request) is often slow but NewRelic reports that it is slow because of "AcquireRequestState". Now I've looked on google and stack overflow and this event is connected to creating a Session but we have sessions disabled in web.config. Does anyone know what else could "AcquireRequestState" could be doing? Are we missing some place to remove session state? Adding that from web.config to machine.config did nothing...

Here is a snippet from a request in NewRelic:

   **Slowest components   Count Duration     %   **
     AcquireRequestState    1   12,600 ms   100%  --> WTF?
     ExecuteRequestHandler  1   5.01 ms     0%
     Integrated Pipeline    1   0.334 ms    0%
     UpdateRequestCache     1   0.3 ms      0%
     EndRequest             1   0.168 ms    0%
     AuthenticateRequest    1   0.161 ms    0%
     Total time                 12,600 ms   100%

EDIT: I have <sessionState mode="Off" /> in web.config (<system.web> section) so that is not it.

Behoof

IINM, you're not loading the modules, but Session is still "enabled" (default) - re: <sessionState mode="Off"> in system.web Hth... – Duplication

I have <sessionState mode="Off" /> in web.config so that is not it. I will edit this to add that info – Behoof

Is this browser specific (IE only) or for all? ... Found this article: forums.iis.net/t/1169137.aspx – Revolutionary

Do you have this set in web.config?: <modules runAllManagedModulesForAllRequests="true"> – Revolutionary

Hi LGSon, no we don't have that line. I know what it does and certainly we would not put it there :), also about the browser - while I don't know what browser was that, the link is specific to .NET 2 so I guess it does not apply to us since we have .NET 4.5.2., tnx for the help anyway ;) – Behoof

Did you try to apply a hotfix mentioned in this question #30067425 – Piggott

Michal we have Windows2012 and that hotfix does not apply to us... – Behoof

@Behoof Have you found a cause of the problem? We have exact the same problem. – Indole

@Indole No... Still the same. Our thinking is that either Newrelic reports wrongly about which component is slowing the system down or it is due to async queuing inside iis (although we do not see this in profile counters). – Behoof

@Behoof Have there been any more developments as we are getting about 10 of these per day as shown in new relic. Similar load as you, but can't figure out what could be causing these few outliers. – Teutonic

Perhaps you can try using a profiler tool to profile your app and see if AcquireSessionState does shows up in the profiler? Jetbrain's profiler – Concavoconvex

@DavidGreenwell @Behoof You need to find out what AcquireRequestState is actually calling. Its just an eventhandler. Once you know what its actually calling, you can find out what that is and where its signing up for the event and, hopefully, how to stop it. – Cribbage

Answer 1

I've looked into this because we had similar issues, I found this forum post, the reply that is interesting is this:

Unfortunately, the issue is not as simple as just turning sessionState off. In fact, one of the key phrases when describing the challenges with AcquireRequestState is the phrase for example, session state when it comes to when this event is raised. In digging deeper into this (actually looking at the .NET source) we can see that this is called when an EventHandler is executed or a RequestNotification object is created. I daresay there are other methods and/or events that, when called, will raise an AcquireRequestState event. Tracking down all of them represents something of a struggle. It seems this is something not talked about a lot outside of the more normalized session state discussions. The most common place we see this event raised is certainly related to session state management. But there are very obviously outliers where these event can still be raised. It can even be called directly from application code. The thing the agent grapples with is that it can identify the event, but rarely the source. When it is raised as part of the ASP pipeline, the only notification the agent gets is that this is one segment of the transaction. The source, or the methods executed inside of the event, is something the agent is rarely instrumenting by default. I wish we could offer more insight for you on this. There are a lot of moving parts inside of a .NET application, some of which involve the operating system itself, IIS, the version of .NET, whether or not the methods are asynchronous, application pool settings, execution modes, permissions, etc. While I don't want to open a second can of worms here, this harkens to the issue with the lack of stack traces for 500 errors. The agent provides a stack trace when it is offered and/or available. Where the stack trace, if one even exists, occurs within the ASP pipeline is extremely important. Sometimes it occurs before any actual application code is executed. In such cases the error is reported to the application, which in turn lets the .NET agent see and report that an error occurred, but no other details are provided. The agent simply sees that it happened and reports as much information as is possible. Beyond that the agent simply has no further details it can offer.

We gave up, so I'd be interested to know if you figure it out!

Asante

Answer 2

AcquireRequestState is also used by the Profile Module. Did you try disabling that?

<modules>
   <remove others unwanted modules... />
   <remove name="Profile" />
</modules>

Glory

Answer 3

So I had this same issue. I had a lot of concurrent ajax requests that all used variables stored in ASP.NET ClaimsIdentity, which is effectlively using the SessionState. This meant, that a lot of the server response time was just waiting for the SessionState to be unlocked.

I solved it by decreasing the Lock Polling interval:

var sessionStateModuleType = typeof(SessionStateModule);
var pollingIntervalFieldInfo = sessionStateModuleType.GetField("LOCKED_ITEM_POLLING_INTERVAL", BindingFlags.NonPublic | BindingFlags.Static);
pollingIntervalFieldInfo.SetValue(null, 30); // default 500ms
var pollingDeltaFieldInfo = sessionStateModuleType.GetField("LOCKED_ITEM_POLLING_DELTA", BindingFlags.NonPublic | BindingFlags.Static);
pollingDeltaFieldInfo.SetValue(null, TimeSpan.FromMilliseconds(15.0)); // default 250ms

The solution above was disclosed in this blog article

Tenno