C3p0 APPARENT DEADLOCK exception

Question

I keep getting this exception in my Tomcat log:

com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector run
WARNING: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@76b28200 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector run
WARNING: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@76b28200 -- APPARENT DEADLOCK!!! Complete Status: 
    Managed Threads: 3
    Active Threads: 3
    Active Tasks: 
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1201fd18 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@408f3be4 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@7ba516d8 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
    Pending Tasks: 
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@137efe53
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@766b0524
Pool thread stack traces:
    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
        java.lang.Thread.sleep(Native Method)
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
        java.lang.Thread.sleep(Native Method)
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
        java.lang.Thread.sleep(Native Method)
        com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)

I am using Hibernate 3.6.2 and C3P0 0.9.1.2 with MySQL. After a couple of hours of searching Google, this APPARENT DEADLOCK exception seems to be usually associated with prepared statement caching. This is my C3P0 configuration in my hibernate.cfg.xml:

<propertyname="connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>
<property name="hibernate.c3p0.acquire_increment">5</property>
<property name="hibernate.c3p0.min_size">5</property>
<property name="hibernate.c3p0.max_size">60</property>
<property name="hibernate.c3p0.idle_test_period">120</property>
<property name="hibernate.c3p0.timeout">180</property>              
<property name="hibernate.c3p0.max_statements">0</property>

I am not doing any statement caching whatsoever. Any hints of what is wrong here would be greatly appreciated.

Toland

Answer 1

The tasks that are dealocking are Connection acquisition tasks. That is, c3p0 is trying to acquire new Connections from your database, and those Connection acquisition attempts are taking a long time.

The first thing I would do is upgrade to 0.9.2.1, which has a much improved means of performing a round of Connection acquisitions in situations where acquisition attempts sometimes fail.

If that doesn't solve your problem, then you'll need to figure out why c3p0's attempts to acquire a Connection are hanging for long periods of time: neither succeeding nor failing with an Exception.

Cannibal

Seems upgrading my c3p0 to version 0.9.2.1 did the trick! Thanks @SteveWaldman for the tip. – Toland

Apparently it didn't do the trick, APPARENT DEADLOCK is back... Is there a way to monitor c3p0's activities closer or at a lower level to see why exactly it is hanging when attempting to aquire a connection? – Toland

Could deleting the cache of the webserver (tomcat/work/Catalina) have anything to do with this? I've found some threads in which people claim this solved their issues. – Toland

tomcat's hot redeploy is rife with landmines for permgen memory leaks with libraries like c3p0 that spawn new Threads. c3p0-0.9.5-pre4, whose release is imminent, perhaps even today, will have some new settings to prevent those leaks, set new config params privilegeSpawnedThreads to true and contextClassLoaderSource to library. maybe wait a day or two and give that a try! – Cannibal

I totally agree with Steve, tomcat's hot redeploy is really close to a "russian roulette"-like option. If you had any luck with the options suggested, please, tell us the results, I'm really curious about. – Diaphone

In our case it was our server was being blocked from communicating with MySQL server. com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. – Highsounding

Answer 2

In my very particular case the problem had to do with the configuration of the server where I deployed my application.

It was only after printing out the stack trace of my application that it was hung because the oracle driver was waiting for a secure random number being generated:

Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
  java.io.FileInputStream.readBytes(Native Method)
  java.io.FileInputStream.read(FileInputStream.java:255)
  sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedBytes(SeedGenerator.java:539)
  sun.security.provider.SeedGenerator.generateSeed(SeedGenerator.java:144)
  sun.security.provider.SecureRandom$SeederHolder.(SecureRandom.java:203)
  sun.security.provider.SecureRandom.engineNextBytes(SecureRandom.java:221)
  java.security.SecureRandom.nextBytes(SecureRandom.java:468)
  oracle.security.o5logon.O5Logon.a(Unknown Source)
  oracle.security.o5logon.O5Logon.(Unknown Source)
  oracle.jdbc.driver.T4CTTIoauthenticate.(T4CTTIoauthenticate.java:582)
  oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:401)
  oracle.jdbc.driver.PhysicalConnection.(PhysicalConnection.java:553)
  oracle.jdbc.driver.T4CConnection.(T4CConnection.java:254)
  oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
  oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:528)
  com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134)
  com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
  com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
  com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
  com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
  com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
  com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
  com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)

Then I had to learn and understand a bit about entropy noise and could realize, with a little help from my Unix friends, that one of the servers had no installed a package that generates the "noise".

As mentioned in the following post I ran the next commands: cat /proc/sys/kernel/random/entropy_avail 23 cat /proc/sys/kernel/random/poolsize 4096

So there where too few entropy available to generate a secure random number.

After installing the package cat /proc/sys/kernel/random/entropy_avail 4096 Afterwards I had no more apparent deadlocks, my application could get connections to the DB.

http://www.usn-it.de/index.php/2009/02/20/oracle-11g-jdbc-driver-hangs-blocked-by-devrandom-entropy-pool-empty/

Patch

Answer 3

Got a similar error like this before.

It could also be caused by database server blocking your IP. Make sure your servers' IP are not blocked by the database server/cloud provider.

In my case, was trying to insert data to the database in the cloud with multithreading, and it only works on the local, but not on a server (on a different network).

Arduous