The state server has closed an expired TCP/IP connection

Question

We are getting 500: Server Error page on our IIS 8.5 server, on different frequency of times, and the number of error is increasing and decreasing with the traffic level.

After investigating Event Log we find below error.

The state server has closed an expired TCP/IP connection. The IP address of the client is 127.0.0.1. The expired Read operation began at 10/11/2015 09:45:50.

(The datetime part varies according to the time it happened)

We tried rectifying issues by following many of the recommended solutions online but nothing could really help us.

The ASPNET_State Service is timing out and hence all the connections to that queue are being abandoned, hence responding with a Server error page.

I've tried increasing the sessionState timeout in web configuration, checked for any memory leak and so nothing has worked so far.

The application is running on ASP.NET 4.0 classic mode and IIS version is IIS 8.5.

Edit 1

I can see multiple requests in the queue dying at the exact same time:

If you refer to the image they're more than 10 requests abandoned at the exact same time!

Also, I can see another error:

An error occurred in while processing a request in state server. Major callstack: ContinueReading-->ProcessReading-->Tracker::Read. Error code: 0x80072736

but that is very rare!

I appreciate any help from experts! Thanks a lot in advance!!

update

How did I get over the problem

minimizing the usage of ViewState to almost zero helped me get over this problem. ASP State Service was not able to handle X number ViewState lookup. So, make it stateless as much as possible.

Corpuscle

Is your server under very heavy load when the timeouts occur? Have you tried modifying the stateNetworkTimeout - defaults to 10 seconds. – Eupepsia

We have set it to 60 seconds and still coming... the number of abandoned requests grows at peak hours (say 1000 users) and comes down when it is low as much as 100 users! The server looks almost free even in Peak hours! it has 32 GB of RAM in which 20% is like always free! – Corpuscle

With the timeout set to 60 seconds do you see the time difference between when the expired read operation began and the time that the event is logged being greater than 60 seconds? In the example above it looks like 38 seconds time difference. – Eupepsia

true! yes, so what's happening is: if more requests are coming to the same queue at the moment it is dying, all the requests in the queue die together. So I can see multiple requests died at the exact same time. – Corpuscle

@Eupepsia please see my edit! – Corpuscle

What's CPU and network load on both machines at that time? – Strappado

@Strappado CPU is like 15% -40% in variation all the time and not more than that! It is just one machine by the way. – Corpuscle

@Strappado however, memory usage is at 80-90% all the time due to SQL Server 2014 is running on the same server – Corpuscle

@Sunny Did you find a solution for this problem? – Labaw

@Labaw minimizing the usage of ViewState to almost zero helped me get over this problem. ASP State Service was not able to handle X number ViewState lookup. Make it stateless as much as possible - learnt as a lesson out of it, – Corpuscle