Tomcat doesn't stop. How can I debug this?

Question

I have a Tomcat 7 running in Linux that I start via $CATALINA_HOME/bin/startup.sh and shutdown via $CATALINA_HOME/bin/shutdown.sh
from /etc/init.d

All is ok except 1 problem. Sometimes tomcat does not stop.
Although I stop it and I see in catalina.out logs that is going down, if I do ps -ef I can still see the process running.

What could be the problem? How can I debug this? My feeling is, that this is related to threads.

So the parts that are suspicious are the following:
1) I use Log4j's LogManager to detect if the log4j configuration has been changed, but I do Log4jManager.shutdown on a contextDestroyed ServletContextListener
2) I use H2 database and I see on shutdown:

SEVERE: The web application [/MyApplication] appears to have started a
thread named [H2 Log Writer MYAPPLICATION] but has failed to stop it.
This is very likely to create a memory leak

SEVERE: The web application [/MyApplication] appears to have started a
thread named [H2 File Lock Watchdog
/opt/myOrg/tomcat/webapps/MyApplication/db/myDatabase.lock.db] but has
failed to stop it. This is very likely to create a memory leak. Apr 2,
2012 9:08:08 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads SEVERE: The web application [/MyApplication]
appears to have started a thread named [FileWatchdog] but has failed
to stop it. This is very likely to create a memory leak.

Any help please? How can I detect the problem here?

UPDATE:
I did a kill -3 as suggested by @daveb, and in the catalina.out I see:

JVMDUMP006I Processing dump event "user", detail "" - please wait. JVMDUMP032I JVM requested Java dump using '/etc/init.d/javacore.20120402.093922.2568.0001.txt' in response to an event JVMDUMP010I Java dump written to /etc/init.d/javacore.20120402.093922.2568.0001.txt JVMDUMP013I Processed dump event "user", detail "".

There is a javacore in /etc/init.d but I don't know how to process it. I.e. what parts should I investigate

Radium

Try marking your threads as daemon threads so that the VM doesn't wait for them to die. docs.oracle.com/javase/6/docs/api/java/lang/… But of course that's only useful for your own threads, not the ones started by H2 – Calaverite

H2 only creates daemon threads. – Passim

@ThomasMueller:So what does the SEVERE: The web application [/MYAPPLICATION] appears to have started a thread named [H2 File Lock Watchdog /opt/myOrg/tomcat/webapps/MyApplication/lock.db] but has failed to stop it. This is very likely to create a memory leak. mean in catalina.out?I didn't create this, but H2. – Radium

This is still a daemon thread. The fact that the thread was not stopped just means the database was not yet closed. The database is closed if all connections to the database as closed, or if you execute the statement "shutdown". – Passim

@ThomasMueller:But my connections are from Tomcat's connection pool.So Tomcat is responsible to close them.I always close them but Tomcat will place them but in the pool.Is this a known issue?How can I resolve this?Should I be doing "shutdown" myself?I also have this post on this #9972872 – Radium

It kind of sounds like Tomcat doesn't dispose the connection pool... I think Tomcat should first dispose the connection pool, and after that log threads that are still running. So in my view it a problem that could be fixed in Tomcat, but not in H2. – Passim

@ThomasMueller:But if this is a Tomcat issue, why does this problem occur only with H2? – Radium

What other databases did you test with? – Passim

I have used MySQL an MS-SQL in the past and never had such subtleties.I am not trying to "blame" H2.I am just trying to see how to get around this. – Radium

@ThomasMueller:I have contact Tomcat guys for this issue and they say that the problem is that the H2 driver is using the web app class loader as the context class loader when you should have the class loader used to load the JDBC. This creates a memory leak driver – Radium

@Radium Thanks! I don't fully understand all this yet, but I will look into that. – Passim

@ThomasMueller:If you want I can send you the mail link of the discussion with the Tomcat dev.I am not sure where do you accept issues of H2 in your H2 site – Radium

@Radium you can send a mail to the H2 Google Group, create an issue in Google Code (project "h2database"), or let me know here - all is fine. The mail link to the discussion would be nice! – Passim

@ThomasMuller:I added a post in your Google Group.It says it is under moderation.By my mistake I didn't add a link to the discussion with Tomcat dev which is here:mail-archives.apache.org/mod_mbox/tomcat-users/201204.mbox/… I will update it though – Radium

@ThomasMueller:Here also mail-archives.apache.org/mod_mbox/tomcat-users/201204.mbox/… – Radium

@ThomasMueller:I have send you a sample reproducing the problem – Radium

jstack and jvisualvm are useful tools for diagnosing such problems – Disproof

Answer 1

If the web application is stopped, all connections to the database should be closed as well. If you don't have a list of connections, then execute the SQL statement "shutdown" (this only works for the H2 and HSQLDB databases).

If you have a registered a Servlet, you can do that in the Servlet.destroy() method.

If you have registered a ServletContextListener, you can execute the "shutdown" statement in the ServletContextListener.contextDestroyed(ServletContextEvent servletContextEvent) method. This is what org.h2.server.web.DbStarter ServletContextListener does (the one that is included in the H2 database).

Passim

Thanks for the reply!If the web application is stopped, all connections to the database should be closed as well. I am using Tomcat's connection pool. So it is out of my hands.Is this a known 'racing' issue between Tomcat and H2.I can do SHUTDOWN (it is safe right?) but I want to make sure I address this correctly – Radium

It see the problem. Not sure what the best solution is... ignore the exception? The statement SHUTDOWN will only close this database, so it should be relatively save - only if you are not sure if other web apps are using the database, then you can't do that. Another solution is to use the server mode (run the H2 database in another process). – Passim

:No the database is mine i.e. no other applications will access it.And I am required to use it in file mode.So basically you are saying that 1) shutdown is safe for the data 2)I can ignore the exception meaning what?That Tomcat does not hang on shutdown because of H2 still running?I see the db.lock file in the directory of my app – Radium

The <databaseName>.lock.db file is not deleted as long as the database is open. The database is open because there is at least one open connection. There is at least one open connection because the connection pool is not disposed. If you execute shutdown then the database is closed. In your case, that is save. – Passim

In my case i have to stop all the connection including quartz thread when tomact stops.Even i have ServletContextListener.contextDestroyed method but i dont know how to stop all connections when tomcat stops – Palaeobotany

Answer 2

Find out what threads are still running (or blocked, waiting to run) by using jstack or sending a signal to the process:

kill -3 pid

When you know this, you can make whatever it was that started them hook into the shutdown notification to stop the threads. Or make those threads deamon threads.

See This tomcat shutdown question for more details on this.

If you don't know where your threads were created, then consider adding names to them - executors can take thread factories, and you can use those factories to set the deamon status of a thread and also to name it - so your stack trace will be clearer.

Shelves

Open the javacore .txt file in a text editor, look for thread stacks that include packages that you've written. – Shelves

Yes I figured that.But I am not sure on what indicates an error. – Radium

Answer 3

I had the exact same problem. Sometimes, the command ./shutdown.sh does not stop the tomcat process, and its java process stays in the running processes.

I had solved this problem using the Tomcat version in the Ubuntu's software repositories, by:

sudo apt-get install tomcat7

After installing it from package manager and configuring some settings, I did not have any problems on stopping/starting Tomcat. I used this command to stop, and it never failed:

service tomcat7 stop

which is nearly the same as

/etc/init.d/tomcat7 stop

Using this command runs the code block from the init script, specifically, the codes from the file /etc/init.d/tomcat7. So I looked into it to see what it does to always kill the tomcat process succesfully. Here is the code block that runs when you use service tomcat7 stop command:

log_daemon_msg "Stopping $DESC" "$NAME"

        set +e
        if [ -f "$CATALINA_PID" ]; then
                start-stop-daemon --stop --pidfile "$CATALINA_PID" \
                        --user "$TOMCAT7_USER" \
                        --retry=TERM/20/KILL/5 >/dev/null
                if [ $? -eq 1 ]; then
                        log_progress_msg "$DESC is not running but pid file exists, cleaning up"
                elif [ $? -eq 3 ]; then
                        PID="`cat $CATALINA_PID`"
                        log_failure_msg "Failed to stop $NAME (pid $PID)"
                        exit 1
                fi
                rm -f "$CATALINA_PID"
                rm -rf "$JVM_TMP"
        else
                log_progress_msg "(not running)"
        fi
        log_end_msg 0
        set -e
        ;;

The important part is this:

start-stop-daemon --stop --pidfile "$CATALINA_PID" \
                            --user "$TOMCAT7_USER" \
                            --retry=TERM/20/KILL/5 >/dev/null

This means "retry stopping until the process is stopped. Here is the --retry command documentation from start-stop-daemon manual:

   -R|--retry timeout|schedule
          With  --stop,  specifies  that  start-stop-daemon  is  to  check
          whether  the  process(es)  do  finish.  It will check repeatedly
          whether any matching processes are running, until none are.   If
          the  processes  do  not exit it will then take further action as
          determined by the schedule.

          If timeout is specified instead of schedule  then  the  schedule
          signal/timeout/KILL/timeout  is used, where signal is the signal
          specified with --signal.
          ...

So, --retry=TERM/20/KILL/5 means "Send TERM signal to the process, wait 20 seconds, if it's still running, send KILL signal, wait 5 seconds, if it's still running, there is a problem.

This means you can configure the tomcat to run as a deamon and use a command like this, or write a script to do that kind of action to stop tomcat, or just use Ubuntu and get the tomcat from the package manager.

Beetner

Answer 4

Check if your Web Application has some Scheduler active, like Quartz.

If you don't stop it, Web Application Thread never ending until you kill it

Grindstone

Answer 5

I also had the same problem. There was a ThrottledThreadPoolExecutor in my application that wasn't getting shutdown. When I shut it down properly, tomcat would stop cleanly. In order to figure out the problem, I had to remove all the apps from my tomcat webapps directory and then add them one by one and see which one was causing the problem

Magnet

Answer 6

If you're using a Scheduler or some other entity in your web app, you need to shut it down. Typically you have to use a ServletContextListener to provide the hook to make your shutdown call. A shutdown hook won't work in this case because the JVM is not shutting down (yet). Believe me, I've tried. If your code is in agent code or something outside the container/webapp, then a shutdown hook SHOULD work, though often it's a hair pulling experience to figure out why it's STILL not working. Note, I'm bald.

Garv

there's also this useful thread: serverfault.com/questions/1021145/…. and oh my was I so sure that I was spawning no threads of mine - and yet I really did, I just forgot. I was so sure that I was searching for old bug reports on legacy iBatis that we are using, thinking that it was the culprit - not cleaning up connections and such. was I so wrong. – Juanjuana

Answer 7

In my case I had one rogue JPA EntityManager that wasn't being properly closed after use. Fixed that, and now I can Clean-and-Build again without killing the damn Java process everytime :)

Niehaus