How can I monitor/log Tomcat's thread pool?

Question

I have a Tomcat installation where I suspect the thread pool may be decreasing over time due to threads not being properly released. I get an error in catalina.out when maxthreads is reached, but I would like to log the number of threads in use to a file every five minutes so I can verify this hypothesis. Would anyone please be able to advise how this can be be done?

Also in this installation there is no Tomcat manager, it appears whoever did the original installation deleted the manager webapp for some reason. I'm not sure if manager would be able to do the above or if I can reinstall it without damaging the existing installation? All I really want to do is keep track of the thread pool.

Also, I noticed that maxthreads for Tomcat is 200, but the max number of concurrent connections for Apache is lower (Apache is using mod_proxy and mod_proxy_ajp (AJP 1.3) to feed Tomcat). That seems wrong too, what is the correct relationship between these numbers?

Any help much appreciated :D

Update: Just a quick update to say the direct JMX access worked. However I also had to set Dcom.sun.management.jmxremote.host. I set it to localhost and it worked, however without it no dice. If anyone else has a similar problem trying to enable JMX I recommend you set this value also, even if you are connecting from the local machine. Seems it is required with some versions of Tomcat.

---

Just a quick update to say the direct JMX access worked. However I also had to set Dcom.sun.management.jmxremote.host. I set it to localhost and it worked, however without it no dice. If anyone else has a similar problem trying to enable JMX I recommend you set this value also, even if you are connecting from the local machine. Seems it is required with some versions of Tomcat.

Braid

If you're running on Java 5+ you might be able to monitor the JVMs thread pool using jconsole. – Prescind

I tried to connect to Tomcat using jconsole but it failed (this was trying to connect using the PID of Tomcat on the localhost, not remotely). Do I need to enable JMX to use jconsole? – Braid

AFAIK, in Java 5 you need to enable JMX. As of Java 6 this should be unnecessary for local access. Alternatively there is a Hotspot JVM option, that creates a thread dump from the command line (IIRC it is -XX:+HeapDumpOnCrtlBreak which will create a heap dump and a thread dump) – Prescind

can you show the error message tomcat is giving? tomcat's max threads setting refers to the number of connectors it can use, which essentially is the number of concurrent connections it can handle... so the two numbers you mention should be similar. At least that's my understanding – Neckwear

The error is SEVERE: All threads (200) are currently busy, waiting. Increase max threads or check the servlet status. Also when this happens Tomcat then stops serving pages and gives 503. I think the two numbers should be similar ie MaxClients in Apache is max number simultaneous connections so should this not also be 200? – Braid

Answer 1

Direct JMX access

Try adding this to catalina.sh/bat:

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=5005
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false

UPDATE: Alex P suggest that the following settings might also be required in some situations:

-Dcom.sun.management.jmxremote.host=localhost

This enables remote anonymous JMX connections on port 5005. You may also consider JVisualVM which is much more please and allows to browse JMX via plugin.

What you are looking for is Catalina -> ThreadPool -> http-bio-8080 -> various interesting metrics.

JMX proxy servlet

Easier method might be to use Tomcat's JMX proxy servlet under: http://localhost:8080/manager/jmxproxy. For instance try this query:

$ curl --user tomcat:tomcat http://localhost:8080/manager/jmxproxy?qry=Catalina:name=%22http-bio-8080%22,type=ThreadPool

A little bit of grepping and scripting and you can easily and remotely monitor your application. Note that tomcat:tomcat is the username/password of user having manager-jmx role in conf/tomcat-users.xml.

Infant

Thank you Tomasz I will give this a go. I don't think I can use the proxy servlet as it's under the manager which unfortunately for some reason appears to have been removed from the installation - the admin dir exists but the manager does not. – Braid

I tried using curl/jmxproxy as above but it just gave a 404 so I think the manager is borked, I'm gonna try enabling JMX on an explicit port as suggested above see if that works. Thanks for the help everyone! – Braid

Answer 2

You can deploy jolokia.war and then retrieve mbeans values in JSON (without the manager):

http://localhost:8080/jolokia/read/Catalina:name=*,type=ThreadPool?ignoreErrors=true

If you want only some values (currentThreadsBusy, maxThreads, currentThreadCount, connectionCount):

http://localhost:8080/jolokia/read/Catalina:name=*,type=ThreadPool/currentThreadsBusy,maxThreads,currentThreadCount,connectionCount?ignoreErrors=true

{
    request: {
       mbean: "Catalina:name="http-nio-8080",type=ThreadPool",
       attribute: [
          "currentThreadsBusy",
          "maxThreads",
          "currentThreadCount",
          "connectionCount"
       ],
       type: "read"
    },
    value: {
       currentThreadsBusy: 1,
       connectionCount: 4,
       currentThreadCount: 10,
       maxThreads: 200
    },
    timestamp: 1490396960,
    status: 200
}

Note: This example works on Tomcat7 +.

Forewoman

You can avoid referring to http-nio-8080 or http-bio-8080 by using a wildcard, which will bring all ThreadPools: curl -XGET 'http://localhost:8080/jolokia/read/Catalina:name=*,type=ThreadPool?ignoreErrors=true' – Freeliving

Answer 3

For a more enterprise solution. I have been using New Relic in our production environment.

This provides a graph of the changes to the threadpool over time.

Velez

Great tool. Too bad the pricing is insane – Catercorner

Answer 4

There are cheaper tools out meanwhile: I am using this jar here: https://docs.cyclopsgroup.org/jmxterm You can automate it via shell/batch scripts. I regexed the output and let prometheus poll it for displaying it in grafana.

Hypochondrium