Intermittent delays in System.Web.HttpApplication.BeginRequest(), not SessionState related

Question

We have two web apps (Azure web roles) that both suffer from occasional long delays (40 to 60 seconds) during System.Web.HttpApplication.BeginRequest. We know this because we are using NewRelic to monitor our web apps. The usual culprit is thread agility issues due to ASP.NET's Session State locking mechanism, however we don't use ASP.NET Session State, and on one of the sites we don't use sessions at all.

One app is much more complex than the other and suffers more delays, but I'll use the simple app in this question to hopefully narrow down the root cause.

The simple web app is a series of ServiceStack based web services. It does not use sessions. It only acts as intermediary to a WCF based service layer. It's mainly channeling the requests to WCF services and then mapping the response to views to transfer back to the agent. The servers don't even break a sweat at the loads they are running (max 2.5% CPU).

So, what is the likely cause please?

1. My best guess is that it is a thread agility issue as it seems to be waiting for something, which would suggest a lock somewhere. But what is it waiting for if not Session State? NewRelic or ServiceStack are causing the lock?
2. NewRelic's reporting is wrong, and there is no problem. Well, NewRelic correctly reported problems when we used to use ASP.NET Session State and were getting a lot more of these delays.

Moonstone

we are having the exact same problem. What was your issue and how did you solve it? Thanks – Strawworm

Answer 1

New Relic doesn't have instrumentation specific to ServiceStack and WCF is very basic without custom instrumentation. Without more information, it's very difficult to offer any advice. Thread agility is also a primary culprit and I'd recommend investigating that route first.

It is possible New Relic is attributing the time to a method it shouldn't. I would probably start by opening a ticket with New Relic Support and include all your info (agent logs, IIS/ASP.NET configuration, custom handlers? and permalinks to your New Relic graphs).

Jeer

Thanks, that sounds like a sensible starting point. I'll see if NewRelic can help be find the solution. I won't mark the answer yet in-case someone can come along with some specific potential causes. – Moonstone

We've found that unless you put in New Relic custom instrumentation for the affected pages New Relic tends to lump things under BeginRequest/BeginProcessRequest. This is because this is the last function New Relic Agent has built in instrumentation for, and therefore any delays after this are considered part of this function as far as the New Relic agent is concerned. To quote New Relic: "it could be that as far as the agent is concerned this is all it know about the transaction. So therefore we are going to report the time in that method." – Moonstone