Tomcat Session Eviction to Avoid OutOfMemoryError

Question

We are running a vendor-supplied webapp in Tomcat 5.5 using the StandardManager for sessions (in memory). As sessions can get quite large (20M+), running out of heap space is a serious concern. Users want to keep sessions around for a couple of hours if possible but would rather evict sessions than run out of heap space. It does not appear that the vendor properly implemented Serializable in session'ed objects, so switching to the persistent session manager implementation isn't an option.

Tomcat allows one to set a maxActiveSessions property that will restrict the overall number of sessions in the manager. However, when that limit is reached, no new sessions may be created until some existing sessions expire. We want to clobber the least recently used sessions first.

Ideally, we would like to expire some not recently used sessions when heap usage approaches the "Xmx" setting even if they are not old enough to expire unconditionally. A very old Tomcat developer mailing list thread suggested that this might allow a denial of service attack*, but, as this application is only available within the corporate network, we are unconcerned.

I thought about extending StandardManager to override processExpires() and clobber additional sessions if heap usage is greater than, say, 85% of max. However, this seems a bit problematic in practice. What if muuch of the heap is unreferenced, and the garbage collector would be able to collect a ton of objects (if it bothered to run) to reduce heap to 50% of max? I'd be expiring sessions unnecessarily. I guess we could mitigate this risk with some aggressive garbage collection settings. Also, how would I know how much memory I had saved by expiring a session? I would have to wait for a few GC cycles to know for sure. Perhaps I could take a conservative approach and remove at most N sessions per background process cycle until memory drops below an acceptable threshold. I could serialize the session as a way to estimate how much stuff would be GC'ed, but that relies on the vendor implementing Serializable and marking instance variables as transient appropriately.

Has anyone solved this problem? As a short-term fix, we are increasing heap size, but that band-aid has its drawbacks as well.

• They were referring to a public site where sessions would be created pre-login. Someone could cause many new sessions to be created and crowd out the ones actually in use.

Update: We really don't have much control over the system's architecture, and we specifically can't reduce session use. However, we can futz with the container as much as we'd like. Tomcat is the only open source servlet container that's vendor-supported, however.

Pedlar

Answer 1

Your options seem to be:

1. reduce the idle session timeout,
2. make sessions persistent (maybe only after the user has logged in),
3. reduce the memory used by each session object,
4. increase the memory for your Tomcat instance,
5. run multiple instances of your service, and put a load balancer in front of it/them.

From a technical standpoint 3 is the best solution ... if it works. The others all have a down-side.

Doing clever things with memory is only a band-aid. From the users' perspective, it makes your site's behaviour harder to understand. Besides, if your user base / traffic is trending upwards, you are only putting off the problem of finding a sustainable solution.

Sollows

Unfortunately, we don't have control over the system's architecture -- it's a vendor-supplied app that we can only customize a bit. However. we do have control over the deployment container. We did increase heap size today, but it's really just a stop-gap change – Pedlar

Sounds like you should be beating on your vendor for a fix to reduce session object usage ... or changing vendor. I guess option 5 might also be feasible. Indeed, if your growth rate is high enough, that will be your only option in the long run. – Sollows

Answer 2

Have you tried increasing the max heap size of the JVM?

The default, if not specified, is only 64mb - which I would say is on the small side for most intensive/full-blown web applications.

The best way to do this with Tomcat is to add the following to setenv.bat/.sh:

export CATALINA_OPTS=-Xmx128m

(substitute whatever value you want for 128 if you'd like larger than 128mb. Also change to correct syntax for Windows / your shell)

The startup and catalina shell scripts for Tomcat have built-in logic to invoke this file if it exists. This is the "best practice" way to specify any custom environment properties you need to set for your Tomcat install - putting the properties in this file is better than editing startup.sh or catalina.sh directly because this file can be portable between Tomcat installations/versions.

You might also be interested in this link: 6 Common Errors in Setting Java Heap Size (which also has a section at the end on How to set java heap size in Tomcat?).

Rintoul

We just upped the heap from 1G to 1.5G. With 20M sessions, we would have felt the pain long ago had we run Tomcat with default JVM settings. – Pedlar

misread your original question, sorry - didn't realize that you had already attempted to increase heap size – Rintoul

Is setting up additional Tomcat instances to form a cluster an option? – Rintoul

I fail on reading comprehension - an app which does not make sure that any data put into the session is Serializable will fail immediately when clustered – Rintoul

@matt b: not necessarily. You could in theory arrange that the load balancer dispatches to the right machine based on a session key. But that's ugly. – Sollows

That scenario also offers no possibility for failover. But I say that it will fail immediately because I believe Tomcat will throw an exception when a distributable app puts a non-Serializable object into Session. – Rintoul

@StephenC @MattB: We load balance already via a hardware load balancer which does sticky-session round robin. We can add more hardware.memory to make the problem less likely to occur. However, it seems like Tomcat ought to implement some sort of "survival mode." – Pedlar

@MattB: True about fail-over. However, the vendor doesn't support failover. When a server dies, users must log into another instance! [No, I don't like this either!] – Pedlar

@MattB: Tomcat seems to have a "distributable" param that, if set to true, will cause it to check for serializability of session objects. tomcat.apache.org/tomcat-5.5-doc/cluster-howto.html By default, it is false, so there are no checks happening. – Pedlar

That "survival mode" you mention is known as a cluster. In a cluster, (basically) each Tomcat instance shares session data with other instances in the cluster so that if any node fails, users can be sent to a different node in the cluster (by the load balancer) and continue their sessions uninterrupted. This is related to the distributable attribute I mentioned earlier. But again, if you can't use a cluster of Tomcat instances, this is a pretty huge tangent which won't help you. – Rintoul

Answer 3

I agree with Stephen C that it's your vendor you need to take up the problem with - as he said even switch vendors. One thing that no one has mentioned here (surprisingly) is that the vendor should also look at cleaning unused session objects.

It's very easy, unless you keep it in mind all the time, to let session size bloat out - to let an object's size bloat out unnoticed - then have large session attribs when we have a list of these objects - then never cleaning them out - if the app is large where lists of objects A, B, C, D, E, F, are displayed in certain sections of the app, but never cleaned out then that's a fundamental housekeeping problem that the vendor has not addressed.

e.g. is there a 'central screen' in the webapp where you navigate to other parts of the app? If so then the vendor should be cleaning up when entering this screen all the objects which would have been collected and stuffed into session on the screens that are accessible from the central page/screen/portal

Edit And use pagination, not load whole lists meeting criteria (except when pagination is part of the criteria)

I hope that comment helps you, and others.

Schoenfelder

Answer 4

I'd advise to front multiple tomcat instances with apache and then use mod_jk to load balance between them.

You can do this without any real clustering so session sharing won't be an issue.

Mod_jk is rock solid and even offers a simple gui to mark instances as out of use etc.

This would also bring many other benefits in terms of resilience etc. I've personally used this setup on a very large scale public facing site and it worked a charm.

The ideal situation is to setup true session sharing but in some cases this is overkill.

See here:

http://tomcat.apache.org/connectors-doc/generic_howto/quick.html

Swartz

We load balance today and could certainly add hardware/instances. I'd just like Tomcat to provide a safety net. – Pedlar