ASP.NET site sometimes freezing up and/or showing odd text at top of the page while loading, on load balanced servers

Question

I have various servers (dev, 2 x test, 2 x prod) running the same asp.net site.

The test and prod servers are in load-balanced pairs (prod1 with prod2, and test1 with test2).

The test server pair is exhibiting some kind of (super) slowdown or freezing during about one in ten page loads. Sometimes a line of text appears at the very top of the page which looks something like:

00 OK Date: Thu, 01 Apr 2010 01:50:09 GMT Server: Microsoft-IIS/6.0 X-Powered_By: ASP.NET X-AspNet-Version:2.0.50727 Cache-Control:private Content-Type:text/html; charset=ut

(the beginning and end are "cut off".)

Has anyone seen anything like this before? Any idea what it means or what's causing it?

Edit: I often see this too when clicking something - it comes up as red text on a yellow page:

XML Parsing Error: not well-formed
Location: http://203.111.46.211/3DSS/CompanyCompliance.aspx?cid=14
Line Number 1, Column 24:2mMTehON9OUNKySVaJ3ROpN" />
                         -----------------------^

If I go back and click again, it works (I see the page I clicked on, not the above error message).

Update:

...And, instead of the page loading, I sometimes just get a white screen with text like this in black (looks a lot like the above text):

HTTP/1.1 302 Found Date: Wed, 21 Apr 2010 04:53:39 GMT Server: Microsoft-IIS/6.0 X-Powered-By: ASP.NET X-AspNet-Version: 2.0.50727 Location: /3DSS/EditSections.aspx?id=3&siteId=56&sectionId=46 Set-Cookie: .3DSS=A6CAC223D0F2517D77C7C68EEF069ABA85E9HFYV64F&FA4209E2621B8DCE38174AD699C9F0221D30D49E108CAB8A828408CF214549A949501DAFAF59F080375A50162361E4AA94E08874BF0945B2EF; path=/; HttpOnly Cache-Control: private Content-Type: text/html; charset=utf-8 Content-Length: 184

object moved here

Where "here" is a link that points to a URL just like the one I'm requesting, except with an extra folder in it, meaning something like:
http://123.1.2.3/MySite//MySite/Page.aspx?option=1

instead of:
http://123.1.2.3/MySite/Page.aspx?option=1

Update:

A colleague of mine found some info saying it might be because the test servers are running iis in 64 bit (64bit win 2003) (prod servers are 32 bit win 2003).

So we tried telling IIS to use 32 bit:
cscript %SYSTEMDRIVE%\inetpub\adminscripts\adsutil.vbs SET W3SVC/AppPools/Enable32bitAppOnWin64 1

%SYSTEMROOT%\Microsoft.NET\Framework\v2.0.50727\aspnet_regiis.exe -i

(from this MS support page)

But iis stopped working altogether (got "server unavailable" on a white page instead of web sites).

Reversing the above (see the link) didn't work at first either. The ASP.NET tab disappeared from our IIS web site properties and we had to mess around for an hour uninstalling (aspnet_regiis.exe -u) and reinstalling 32 bit ASP.NET and adding Default.aspx manually back into default documents.

We'll probably try again in a few days, if anyone has anything to add in the meantime, please do.

Update: This seems at odds with everything we've found out so far, but our testing shows that this problem happens only in Firefox, not IE or Chrome (!!??).

Update: The Solution
For anyone finding this later:
On Aristos's suggestion (see accepted answer) we searched the code for the HTTP Header "Content-Length". There was a page which set it, a page that pulls an image out of the DB for displaying a company logo (spits it straight to response, i.e.: instead of linking to say "log56.gif" you can link to "ThisImagePage.aspx?id=56" and it will serve the specified gif from the DB).
We commented out the line:

HttpContext.Current.Response.AddHeader("Content-Length", File.Length.ToString());

... and it worked. If anyone can see a bug in that, let us know, otherwise I guess it was some kind of IIS or load balancer configuration problem, that only appears when manually setting the content-length on binary files, and only in Firefox (!?).

Toledo

Is this happening on both the test and live pools? – Jihad

@John Christensen: Just test. – Toledo

Do you use gzip compression on iis - and at the same time you make any gzip inside your asp.net code ? – Allopath

@Aristos: Yes, I think JQuery.js is gzipped. Should I try using the non-gzipped version? – Toledo

I am very glad that help you. :) If some one set something on header and iis is not configure it to have the ability to change, then after iis gzip it probably fails to set correctly the Content-Length and if the browser is count on that length, you get trimmer or false data. – Allopath

Answer 1

MGOwen I will share my experience for a similar problem that I was have.

Some time ago I have a similar problem, the pages work well, except some pages that after compress them with gZipped they have problems and not working correctly, something like yours.

I discover that the problem was because I set the Content-Length on header, and then for some reason when was going to be gZipped the Content-Length, didn't change / or was calculate false, and the result was a similar error like yours.

So check if you set the Content-Length in a way on your pages, and then use gZip filter. If yes then remove the Content-Length setting in your program.

In general speaking I say that the length have the problem on your page, and the lenght is a variable in Content-Length.

-and in your header the Content-Length exist !-

update
Also one other think that I notice, if your page send as gZip where is that in your head ? if this is the full head of your page.

Allopath

Answer 2

are the machines sharing the same machine key?
This along with accessing Session state could cause strange errors.

Brownell

We tried changing them to different session keys, that causes a worse error (the ASP.NET error message tells you to use the same machine key when load balancing) – Toledo

Answer 3

The text you are seeing is the page header. I'm guessing both it and the xml parsing error are being caused by the output to the browser being cut off or, even more bizarrely, only a chunk from the response being relayed.

I'd start with the load balancer and see if there are any logs available. After that, I'd try disabling the IIS compression that Aristos mentioned and see if that has an affect (in IIS get properties on the "Web Sites" folder and then go to the Service tab or find out if compression was enabled / changed for that particular site).

After that, you'll probably have to resort to some kind of packet sniffer to see what is actually being sent on either side of the balancer.

Appreciable

Chose the other answer as that's what led us to the solution, but +1 because this was useful and probably correct. – Toledo

Answer 4

It seems like this issue is with the IIS and .net.

Click on Start ->Run and type in the following command if you are using .NET 2.0 framework: %Windir%\Microsoft.NET\Framework\v2.0.50727\aspnet_regiis -i

Detailed discussion here: http://social.msdn.microsoft.com/Forums/en-US/xmlandnetfx/thread/47f6eb6a-b062-4f4d-8b7f-b4afb1b2725d

Ednaedny

Thanks. Unfortunately, this didn't work for me. The linked discussion seems to be about the XML parsing error screen coming up all the time, rather than occasionally, and the more common symptoms aren't mentioned, so I guess it might be a totally different problem. – Toledo