Right now I am experiencing an unpleasant issue. I have 6 projects running on my JBoss AS 7. They are deployed in standalone mode. After working for some time(5-25 hours) I start to recieve identical error in all my projects : javax.enterprise.event.ObserverException .
Here is the stack:
ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[site.technorational.kz]] (http--192.168.1.2-80-3) Exception sending request initialized lifecycle event to listener instance of class org.jboss.seam.transaction.TransactionServletListener: javax.enterprise.event.ObserverException
at sun.reflect.GeneratedConstructorAccessor136.newInstance(Unknown Source) [:1.6.0_21]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) [rt.jar:1.6.0_21]
at java.lang.reflect.Constructor.newInstance(Constructor.java:513) [rt.jar:1.6.0_21]
at java.lang.Class.newInstance0(Class.java:355) [rt.jar:1.6.0_21]
at java.lang.Class.newInstance(Class.java:308) [rt.jar:1.6.0_21]
at org.jboss.weld.util.reflection.SecureReflections$16.work(SecureReflections.java:343) [weld-core-1.1.5.AS71.Final.jar:2012-02-10 15:31]
at org.jboss.weld.util.reflection.SecureReflectionAccess.run(SecureReflectionAccess.java:52) [weld-core-1.1.5.AS71.Final.jar:2012-02-10 15:31]
at org.jboss.weld.util.reflection.SecureReflectionAccess.runAsInstantiation(SecureReflectionAccess.java:173) [weld-core-1.1.5.AS71.Final.jar:2012-02-10 15:31]
at org.jboss.weld.util.reflection.SecureReflections.newInstance(SecureReflections.java:340) [weld-core-1.1.5.AS71.Final.jar:2012-02-10 15:31]
at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:33) [weld-core-1.1.5.AS71.Final.jar:2012-02-10 15:31]
at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:73) [weld-core-1.1.5.AS71.Final.jar:2012-02-10 15:31]
at org.jboss.weld.injection.MethodInjectionPoint.invokeOnInstanceWithSpecialValue(MethodInjectionPoint.java:162) [weld-core-1.1.5.AS71.Final.jar:2012-02-10 15:31]
at org.jboss.weld.event.ObserverMethodImpl.sendEvent(ObserverMethodImpl.java:241) [weld-core-1.1.5.AS71.Final.jar:2012-02-10 15:31]
at org.jboss.weld.event.ObserverMethodImpl.sendEvent(ObserverMethodImpl.java:229) [weld-core-1.1.5.AS71.Final.jar:2012-02-10 15:31]
at org.jboss.weld.event.ObserverMethodImpl.notify(ObserverMethodImpl.java:207) [weld-core-1.1.5.AS71.Final.jar:2012-02-10 15:31]
at org.jboss.weld.manager.BeanManagerImpl.notifyObservers(BeanManagerImpl.java:569) [weld-core-1.1.5.AS71.Final.jar:2012-02-10 15:31]
at org.jboss.weld.manager.BeanManagerImpl.fireEvent(BeanManagerImpl.java:564) [weld-core-1.1.5.AS71.Final.jar:2012-02-10 15:31]
at org.jboss.weld.event.EventImpl.fire(EventImpl.java:68) [weld-core-1.1.5.AS71.Final.jar:2012-02-10 15:31]
at org.jboss.seam.transaction.TransactionServletListener.requestInitialized(TransactionServletListener.java:117) [seam-transaction-3.1.0.Final.jar:3.1.0.Final]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:]
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877) [jbossweb-7.0.13.Final.jar:]
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671) [jbossweb-7.0.13.Final.jar:]
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:]
at java.lang.Thread.run(Thread.java:619) [rt.jar:1.6.0_21]
Caused by: javax.transaction.NotSupportedException: BaseTransaction.checkTransactionState - ARJUNA016051: thread is already associated with a transaction!
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.begin(BaseTransaction.java:63)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.begin(BaseTransactionManagerDelegate.java:65)
at org.jboss.tm.usertx.client.ServerVMClientUserTransaction.begin(ServerVMClientUserTransaction.java:142)
at org.jboss.seam.transaction.UTTransaction.begin(UTTransaction.java:51) [seam-transaction-3.1.0.Final.jar:3.1.0.Final]
at org.jboss.seam.transaction.DefaultSeamTransaction.begin(DefaultSeamTransaction.java:88) [seam-transaction-3.1.0.Final.jar:3.1.0.Final]
at org.jboss.seam.transaction.TransactionServletListener.requestInitialized(TransactionServletListener.java:110) [seam-transaction-3.1.0.Final.jar:3.1.0.Final]
... 8 more
Caused by: java.lang.IllegalStateException: BaseTransaction.checkTransactionState - ARJUNA016051: thread is already associated with a transaction!
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.checkTransactionState(BaseTransaction.java:257)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.begin(BaseTransaction.java:59)
... 13 more
After I was searching for answers in the logs I found that sometimes this comes before the previous exception comes out:
22:19:21,872 INFO [org.jboss.as.server] (Controller Boot Thread) JBAS018559: Deployed "project1-web.war"
22:19:21,873 INFO [org.jboss.as.server] (Controller Boot Thread) JBAS018559: Deployed "project2-web.war"
22:19:21,873 INFO [org.jboss.as.server] (Controller Boot Thread) JBAS018559: Deployed "project3-web.war"
22:19:21,875 INFO [org.jboss.as.server] (Controller Boot Thread) JBAS018559: Deployed "project4-web.war"
22:19:21,875 INFO [org.jboss.as.server] (Controller Boot Thread) JBAS018559: Deployed "project5-web.war"
22:19:21,876 INFO [org.jboss.as.server] (Controller Boot Thread) JBAS018559: Deployed "project6-web.war"
22:19:21,888 INFO [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://192.168.2.1:9990/management
22:19:21,889 INFO [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://192.168.2.1:9990
22:19:21,889 INFO [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss AS 7.2.0.Alpha1-SNAPSHOT "Steropes" started in 31119ms - Started 4319 of 4426 services (98 services are passive or on-demand)
22:24:59,256 WARN [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffffc0a80107:2c60f317:50914f6c:4d in state RUN
22:24:59,259 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012095: Abort of action id 0:ffffc0a80107:2c60f317:50914f6c:4d invoked while multiple threads active within it.
22:24:59,259 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012108: CheckedAction::check - atomic action 0:ffffc0a80107:2c60f317:50914f6c:4d aborting with 1 threads active!
22:24:59,260 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012121: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 0,5,main] successfully canceled TX 0:ffffc0a80107:2c60f317:50914f6c:4d
I have given a lot of thinking to this issue. One of my suggestions is that MySQL causes that, because there are a lot of methods that are retrieving information from database using JPA CriteriaBuilder. When I check status of MySQL, I can see that there are about 100 slow queries.
In my projects I am using SEAM, Hibernate, JPA, PrimeFaces.
I managed to find a some similar issues but not answers still.
Thanks in advance, Ilya Sidorovich