JBOSS: Thread freezing on Oracle XA Transaction using Atomikos

Question

I am getting an error after sometime on my application when I going to transmit some data using atomikos and hibernate.

2015-11-06 07:11:56,353 WARN [http-/0.0.0.0:8083-10] datasource.xa.XAResourceTransaction - XA resource 'COTXADBMS': resume for XID '31302E3235332E312E35322E746D30303030313030303939:31302E3235332E312E35322E746D31' raised -7: the XA resource has become unavailable - (Slf4jLogger.java:24) 
oracle.jdbc.xa.OracleXAException
        at oracle.jdbc.xa.OracleXAResource.checkError(OracleXAResource.java:1110)
        at oracle.jdbc.xa.client.OracleXAResource.start(OracleXAResource.java:240)
        at com.atomikos.datasource.xa.XAResourceTransaction.resume(XAResourceTransaction.java:427)
        at com.atomikos.datasource.xa.session.BranchEnlistedStateHandler.<init>(BranchEnlistedStateHandler.java:59)
        at com.atomikos.datasource.xa.session.NotInBranchStateHandler.checkEnlistBeforeUse(NotInBranchStateHandler.java:64)
        at com.atomikos.datasource.xa.session.TransactionContext.checkEnlistBeforeUse(TransactionContext.java:88)
        at com.atomikos.datasource.xa.session.SessionHandleState.notifyBeforeUse(SessionHandleState.java:179)
        at com.atomikos.jdbc.AtomikosConnectionProxy.enlist(AtomikosConnectionProxy.java:223)
        at com.atomikos.jdbc.AtomikosConnectionProxy.invoke(AtomikosConnectionProxy.java:142)
        at com.sun.proxy.$Proxy101.prepareStatement(Unknown Source)
        at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:161)
        at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:182)
        at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:159)
        at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1858)
        at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1835)
        at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1815)
        at org.hibernate.loader.Loader.doQuery(Loader.java:899)
        at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:341)
        at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:311)
        at org.hibernate.loader.Loader.loadEntity(Loader.java:2117)
        at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:82)
        at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:72)
        at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:3927)
        at org.hibernate.event.internal.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:460)
        at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:429)
        at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:206)
        at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:262)
        at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:150)
        at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1092)
        at org.hibernate.internal.SessionImpl.access$2000(SessionImpl.java:175)
        at org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.load(SessionImpl.java:2476)
        at org.hibernate.internal.SessionImpl.get(SessionImpl.java:992)
        at org.hibernate.event.internal.DefaultMergeEventListener.entityIsDetached(DefaultMergeEventListener.java:271)
        at org.hibernate.event.internal.DefaultMergeEventListener.onMerge(DefaultMergeEventListener.java:151)
        at org.hibernate.event.internal.DefaultMergeEventListener.onMerge(DefaultMergeEventListener.java:76)
        at org.hibernate.internal.SessionImpl.fireMerge(SessionImpl.java:914)
        at org.hibernate.internal.SessionImpl.merge(SessionImpl.java:898)
        at org.hibernate.internal.SessionImpl.merge(SessionImpl.java:902)
        at org.hibernate.ejb.AbstractEntityManagerImpl.merge(AbstractEntityManagerImpl.java:889)
        at sun.reflect.GeneratedMethodAccessor227.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityManagerInvocationHandler.invoke(ExtendedEntityManagerCreator.java:342)
        at com.sun.proxy.$Proxy112.merge(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor227.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:289)
        at com.sun.proxy.$Proxy112.merge(Unknown Source)
        at br.com.empresa.cotador.projeto.component.EfetivacaoComponent.transmitir(EfetivacaoComponent.java:390)
        at br.com.empresa.cotador.projeto.component.EfetivacaoComponent$$FastClassBySpringCGLIB$$55efafb6.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:708)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:98)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:262)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:95)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:644)
        at br.com.empresa.cotador.projeto.component.EfetivacaoComponent$$EnhancerBySpringCGLIB$$9f2cee0f.transmitir(<generated>)
        at br.com.empresa.cotador.projeto.controller.emissao.EfetivacaoController.transmitirProposta(EfetivacaoController.java:218)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:215)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:749)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:689)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:83)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:938)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:870)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:961)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:863)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:754)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:837)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
        at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:164)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
        at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:202)
        at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:180)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:420)
        at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50)
        at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50)
        at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926)
        at java.lang.Thread.run(Thread.java:745)

And then we catch a "freeze"

"http-/0.0.0.0:8083-2" daemon prio=10 tid=0x0000000001914800 nid=0x7640 runnable [0x0000000045ff8000]
   java.lang.Thread.State: RUNNABLE
            at java.net.SocketInputStream.socketRead0(Native Method)
            at java.net.SocketInputStream.read(SocketInputStream.java:152)
            at java.net.SocketInputStream.read(SocketInputStream.java:122)
            at oracle.net.ns.Packet.receive(Packet.java:300)
            at oracle.net.ns.DataPacket.receive(DataPacket.java:106)
            at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)
            at oracle.net.ns.NetInputStream.read(NetInputStream.java:260)
            at oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
            at oracle.net.ns.NetInputStream.read(NetInputStream.java:102)
            at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
            at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
            at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)
            at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
            at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
            at oracle.jdbc.driver.T4CTTIOtxse.doOTXSE(T4CTTIOtxse.java:163)
            at oracle.jdbc.driver.T4CXAResource.doStart(T4CXAResource.java:186)
            - locked <0x000000078e7f7a40> (a oracle.jdbc.driver.T4CConnection)
            at oracle.jdbc.xa.client.OracleXAResource.start(OracleXAResource.java:228)
            - locked <0x000000078e7f7a40> (a oracle.jdbc.driver.T4CConnection)
            at com.atomikos.datasource.xa.XAResourceTransaction.resume(XAResourceTransaction.java:427)
            - locked <0x00000007ea3500c8> (a com.atomikos.datasource.xa.XAResourceTransaction)
            at com.atomikos.datasource.xa.session.BranchEnlistedStateHandler.<init>(BranchEnlistedStateHandler.java:59)
            at com.atomikos.datasource.xa.session.NotInBranchStateHandler.checkEnlistBeforeUse(NotInBranchStateHandler.java:64)
            at com.atomikos.datasource.xa.session.TransactionContext.checkEnlistBeforeUse(TransactionContext.java:88)
            - locked <0x00000007ea1f2418> (a com.atomikos.datasource.xa.session.TransactionContext)
            at com.atomikos.datasource.xa.session.SessionHandleState.notifyBeforeUse(SessionHandleState.java:179)
            - locked <0x000000078e807730> (a com.atomikos.datasource.xa.session.SessionHandleState)
            at com.atomikos.jdbc.AtomikosConnectionProxy.enlist(AtomikosConnectionProxy.java:223)
            at com.atomikos.jdbc.AtomikosConnectionProxy.invoke(AtomikosConnectionProxy.java:142)
            at com.sun.proxy.$Proxy123.prepareStatement(Unknown Source)
            at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:161)
            at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:182)
            at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:159)
            at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1858)
            at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1835)
            at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1815)
            at org.hibernate.loader.Loader.doQuery(Loader.java:899)
            at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:341)
            at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:311)
            at org.hibernate.loader.Loader.loadEntity(Loader.java:2117)
            at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:82)
            at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:72)
            at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:3927)
            at org.hibernate.event.internal.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:460)
            at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:429)
            at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:206)
            at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:262)
            at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:150)
            at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1092)
            at org.hibernate.internal.SessionImpl.access$2000(SessionImpl.java:175)
            at org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.load(SessionImpl.java:2476)
            at org.hibernate.internal.SessionImpl.get(SessionImpl.java:992)
            at org.hibernate.event.internal.DefaultMergeEventListener.entityIsDetached(DefaultMergeEventListener.java:271)
            at org.hibernate.event.internal.DefaultMergeEventListener.onMerge(DefaultMergeEventListener.java:151)
            at org.hibernate.event.internal.DefaultMergeEventListener.onMerge(DefaultMergeEventListener.java:76)
            at org.hibernate.internal.SessionImpl.fireMerge(SessionImpl.java:914)
            at org.hibernate.internal.SessionImpl.merge(SessionImpl.java:898)
            at org.hibernate.internal.SessionImpl.merge(SessionImpl.java:902)
            at org.hibernate.ejb.AbstractEntityManagerImpl.merge(AbstractEntityManagerImpl.java:889)
            at sun.reflect.GeneratedMethodAccessor256.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:606)
            at org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityManagerInvocationHandler.invoke(ExtendedEntityManagerCreator.java:342)
            at com.sun.proxy.$Proxy138.merge(Unknown Source)
            at sun.reflect.GeneratedMethodAccessor256.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:606)
            at org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:289)
            at com.sun.proxy.$Proxy138.merge(Unknown Source)
            at br.com.empresa.cotador.projeto.component.EfetivacaoComponent.transmitir(EfetivacaoComponent.java:390)
            at br.com.empresa.cotador.projeto.component.EfetivacaoComponent$$FastClassBySpringCGLIB$$55efafb6.invoke(<generated>)
            at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
            at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:708)
            at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
            at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:98)
            at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:262)
            at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:95)
            at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
            at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:644)
            at br.com.empresa.cotador.projeto.component.EfetivacaoComponent$$EnhancerBySpringCGLIB$$4b6893af.transmitir(<generated>)
            at br.com.empresa.cotador.projeto.controller.emissao.EfetivacaoController.transmitirProposta(EfetivacaoController.java:218)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:606)
            at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:215)

...

It happens after some time when the connection is active, and even not using this happens.

Setup - spring applicationXML

<context:component-scan base-package="br.com">
    <context:exclude-filter type="annotation" expression="org.springframework.stereotype.Controller"/>
</context:component-scan>


<bean id="dsDataSourceCotador" class="org.springframework.jndi.JndiObjectFactoryBean">
    <property name="jndiName" value="java:/comp/env/jdbc/cotacaoDS" />
</bean>
<bean id="entityManagerFactoryCotador" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
    <property name="persistenceUnitName" value="puprojetoCotador" />
    <property name="dataSource" ref="dsDataSourceCotador"/>
    <property name="jpaVendorAdapter">
       <bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter" />
    </property>
    <property name="packagesToScan" value="br.com.empresa.cotador.projeto.model" />

    <property name="jpaProperties">
       <props>
          <prop key="hibernate.dialect">org.hibernate.dialect.Oracle9Dialect</prop>
          <prop key="hibernate.show_sql">true</prop>
          <prop key="log4j.logger.org.hibernate.type">TRACE</prop>
          <prop key="connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</prop>
          <prop key="hibernate.c3p0.acquire_increment">1</prop>
          <prop key="hibernate.c3p0.idle_test_period">300</prop>
          <prop key="hibernate.c3p0.timeout">5000</prop>
          <prop key="hibernate.c3p0.max_size">10</prop>
          <prop key="hibernate.c3p0.max_statements">0</prop>
          <prop key="hibernate.c3p0.min_size">1</prop>
          <prop key="hibernate.c3p0.acquireRetryAttempts">3</prop>
          <prop key="hibernate.c3p0.acquireRetryDelay">1000</prop>
          <prop key="hibernate.jdbc.batch_size">30</prop>
       </props>
    </property>
</bean>
<bean id="transactionManagerCotador" class="org.springframework.orm.jpa.JpaTransactionManager">
  <property name="entityManagerFactory" ref="entityManagerFactoryCotador"/>
  <qualifier value="cot"/>
</bean>
<tx:annotation-driven transaction-manager="transactionManagerCotador" proxy-target-class="false"/>



<bean id="activeMQConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
    <property name="brokerURL">
        <value>#{systemProperties['empresa.mr.servidor.ACTIVEMQ']}</value>
    </property>
    <property name="userName">
        <value>#{ systemProperties['empresa.mr.servidor.ACTIVEMQ.USER'] }</value>
    </property>
    <property name="password">
        <value>#{ systemProperties['empresa.mr.servidor.ACTIVEMQ.PASS'] }</value>
    </property>
</bean>

<bean id="connectionFactory"
    class="org.springframework.jms.connection.CachingConnectionFactory">
    <constructor-arg ref="activeMQConnectionFactory" />
</bean>

<bean id="filaRecepcaoMRPut" class="org.apache.activemq.command.ActiveMQQueue">
    <constructor-arg index="0" value="#{systemProperties['empresa.mr.fila.RECEPCAO.PUT']}" />
</bean>

<bean id="filaUploadMRPut" class="org.apache.activemq.command.ActiveMQQueue">
    <constructor-arg index="0" value="#{systemProperties['empresa.mr.fila.UPLOAD.PUT']}" />
</bean>

<bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate">
    <property name="connectionFactory" ref="connectionFactory" />
</bean>



<bean id="queueConnectionFactoryBean" class="com.atomikos.jms.AtomikosConnectionFactoryBean" init-method="init" destroy-method="close">
    <property name="uniqueResourceName" value="QUEUE_BROKER" />
    <property name="xaConnectionFactory" ref="activeMQXAConnectionFactory" />
</bean>

<bean id="jmsTemplateXA" class="org.springframework.jms.core.JmsTemplate">
    <property name="connectionFactory" ref="queueConnectionFactoryBean" />
    <property name="sessionTransacted" value="true"/>
</bean>

<bean id="datasourceCOTXA" class="br.com.empresa.cotador.projeto.util.CustomAtomikosDataSourceBean" init-method="init" destroy-method="close" lazy-init="true">
    <property name="uniqueResourceName" value="COTXADBMS" />
    <property name="wrapperDataSource">
        <bean id="dsDataSourceCOTXA" class="org.springframework.jndi.JndiObjectFactoryBean">
            <property name="jndiName" value="java:/comp/env/jdbc/cotacaoDSXA" />
        </bean>
    </property>
    <qualifier value="cot" />
</bean>

<bean id="entityManagerFactoryCotadorXA" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
    <property name="persistenceUnitName" value="puprojetoCotadorXA" />
    <property name="jtaDataSource" ref="datasourceCOTXA"/>
    <property name="jpaVendorAdapter">
       <bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter" />
    </property>
    <property name="packagesToScan" value="br.com.empresa.cotador.projeto.model" />

    <property name="jpaProperties">
        <props>
            <prop key="hibernate.transaction.manager_lookup_class">com.atomikos.icatch.jta.hibernate3.TransactionManagerLookup</prop>
            <prop key="hibernate.dialect">org.hibernate.dialect.Oracle10gDialect</prop>
            <prop key="hibernate.show_sql">true</prop>
            <prop key="javax.persistence.transactionType">jta</prop>

       </props>
    </property>
</bean>

<bean id="atomikosUserTransactionService" class="com.atomikos.icatch.config.UserTransactionServiceImp" init-method="init" destroy-method="shutdownForce">
    <constructor-arg>
        <props>
            <prop key="com.atomikos.icatch.service">br.com.empresa.cotador.projeto.util.atomikos.UserTransactionServiceFactory</prop>
            <prop key="com.atomikos.icatch.log_base_name">CotadorprojetoServiceXA</prop>
            <prop key="com.atomikos.icatch.output_dir">/app/tmp/</prop>
            <prop key="com.atomikos.icatch.log_base_dir">/app/tmp/</prop>
            <prop key="com.atomikos.icatch.default_jta_timeout">600000</prop>
            <prop key="com.atomikos.icatch.max_timeout">600000</prop>
        </props>
    </constructor-arg>
</bean>

<bean id="transactionManagerXA" class="org.springframework.transaction.jta.JtaTransactionManager">
    <property name="transactionManager">
        <bean id="atomikosTransactionManager" class="com.atomikos.icatch.jta.UserTransactionManager" init-method="init" destroy-method="close">
            <property name="forceShutdown" value="true" />
        </bean>
    </property>
    <property name="userTransaction">
        <bean id="atomikosUserTransaction" class="com.atomikos.icatch.jta.UserTransactionImp" depends-on="atomikosUserTransactionService">
            <property name="transactionTimeout" value="600000" />
        </bean>
    </property>
    <qualifier value="XA"/>
</bean>
<tx:annotation-driven transaction-manager="transactionManagerXA" />

does anonyone whow to avoid this problem, fix or do a workaround?

Encumbrance

A similar problem is discussed in the Atomikos forum. Solution is to set sessionTransacted=true in the JMS template configuration. – Truehearted

Yes, but in this case is really similar, but not the same problem, once on Atomikos website we can see a solution on oracle JMS problem, not a JDBC XA Transaction problem. – Wheen

could you describe your setup? which instances of which software participate in your XA transaction? – Sate

i found the topic on atomikos about the same problem there. Here you can see the topic, but nobody has answered it. fogbugz.atomikos.com/default.asp?community.6.1453.2 – Wheen

Answer 1

I had a similar problem a few years ago using weblogic as transaction manager instead of atomikos. In our case we had aleatory exceptions with oracle in which the transaction had freezed in oracle and got busy the connection object.

We used JTA transactions, I see in your config file that you are using JPA transactions, but i don´t see the persistent file (in which you would configure transaction type as JTA)

The problem was related to timeout transactions, concretely to avoid it:

You have to ensure that any transaction timeout (JTA, JPA, etc...) is lower than the lowest value for XA transactions timeouts (like oracle XA connections).

To achieve this, you can assign in JDBC connection pool configuration the following:

XASetTransactionTimeout -> true 
XATransactionTimeout -> 0

With XATransactionTimeout -> 0 you ensure that the XA timeout will be the transaction manager timeout

Hope helps!

Townsley

I am trying to adapt your answer to JBOSS, looking forward to this link – Wheen

The default values for XATransactionTimeout is 0, but for XASetTransactionTimeout is false, so try to change this value. – Townsley

The problem is just the values, JBOSS there is no XASetTransactionTimeout, and the most approximate values are <set-tx-query-timeout>true</set-tx-query-timeout> and <query-timeout>120</query-timeout> on standalone.xml for TAG <xa-datasource/> so im trying to addapt to these server values. – Wheen

Yep, these are weblogic parameters, but as you said, jboss have to have the corresponding parameters to those from weblogic. Hope you can fix that problem! – Townsley

The problem still continues. <set-tx-query-timeout> didn't work for JBOSS – Wheen

This answer really solves the Weblogic problem, but I really dont know how can be solved on JBOSS AS 6.3 – Wheen

Answer 2

First thing I can say this is not an answer and is the answer, can solve the Thread Freezing problem, but didn't solve the real problem that Thread Freezing was masking. For Thread Freezing problem is in fact the datasource pool max size that differs from standalone.xml on JBOSS. If You create a maxPoolSize on your Spring application different of standalone.xml, atomikos will have problems about requesting and borrow connections.

Look application.xml:

<bean id="datasourceCOTXA" class="br.com.empresa.util.CustomAtomikosDataSourceBean" init-method="init" destroy-method="close" lazy-init="true">
    <property name="uniqueResourceName" value="COTXADBMS" />
    <property name="wrapperDataSource">
        <bean id="dsDataSourceCOTXA" class="org.springframework.jndi.JndiObjectFactoryBean">
            <property name="jndiName" value="java:/comp/env/jdbc/cotacaoDSXA" />
        </bean>
    </property>
    <property name="minPoolSize" value="5" />
    <property name="maxPoolSize" value="50" /><!-- detail here-->
    <property name="testQuery" value="select 1 from dual" />
    <qualifier value="cot" />
</bean>

Now see the datasource on JBOSS:

<xa-datasource enabled="true" use-java-context="true" pool-name="jdbc/cotacaoDSXA" jndi-name="java:/comp/env/jdbc/cotacaoDSXA">
<xa-datasource-property name="URL">... </xa-datasource-property>
<driver>oracleXA</driver>
<xa-pool>
   <min-pool-size>5</min-pool-size>
   <max-pool-size>30</max-pool-size>
    <prefill>true</prefill>
</xa-pool>
...
</xa-datasource>

the thread freezes because atomikos will look for internal pool first, and the max pool size is higher than datasource, the it freezes.

Well, and you will think "So this solves the problem", and my answer is, no, just doesn't solve, but stop masking the problem.

What happens after the Thread freezing problem solved? The problem is between XA and JBOSS 6.2, changing log level of ATOMIKOS you will see some details about what happens with the connection.

ENVIRONMENT A: ATOMIKOS WORKING

2015-12-09 16:14:51,647 INFO [ServerService Thread Pool -- 49] atomikos.logging.LoggerFactory - Using Slf4J for logging. - (Slf4jLogger.java:16) 
2015-12-09 16:14:51,714 INFO [ServerService Thread Pool -- 49] atomikos.jdbc.AbstractDataSourceBean - AtomikosDataSoureBean 'COTXADBMS': init... - (Slf4jLogger.java:16) 
2015-12-09 16:14:51,715 WARN [ServerService Thread Pool -- 49] atomikos.jdbc.AbstractDataSourceBean - AtomikosDataSoureBean 'COTXADBMS': testQuery set - pool may be slower / you might want to consider setting maxLifetime instead... - (Slf4jLogger.java:12) 
2015-12-09 16:14:51,764 INFO [ServerService Thread Pool -- 49] atomikos.jdbc.AtomikosDataSourceBean - AtomikosDataSoureBean 'COTXADBMS': initializing with [ xaDataSourceClassName=oracle.jdbc.xa.client.OracleXADataSource, uniqueResourceName=COTXADBMS, maxPoolSize=30, minPoolSize=5, borrowConnectionTimeout=30, maxIdleTime=60, reapTimeout=0, maintenanceInterval=60, testQuery=select 1 from dual, xaProperties=[URL=jdbc:oracle:thin:@SRVODB003D:1527:COTD,user=APPCOT,password=APCTO234], loginTimeout=0, maxLifetime=0] - (Slf4jLogger.java:16) 
2015-12-09 16:14:52,476 INFO [ServerService Thread Pool -- 49] imp.thread.TaskManager - THREADS: using JDK thread pooling... - (Slf4jLogger.java:16) 
2015-12-09 16:14:52,599 INFO [ServerService Thread Pool -- 49] atomikos.jdbc.AbstractDataSourceBean - AtomikosDataSoureBean 'COTXADBMS': getConnection ( null )... - (Slf4jLogger.java:16) 
2015-12-09 16:14:52,604 INFO [ServerService Thread Pool -- 49] atomikos.jdbc.AbstractDataSourceBean - AtomikosDataSoureBean 'COTXADBMS': init... - (Slf4jLogger.java:16) 

ENVIRONMENT B: ATOMIKOS NOT WORKING AFTER SOME TIME ON JBOSS RUNNING.

 2015-12-11 12:45:39,477 INFO [http-/0.0.0.0:8083-10] atomikos.jdbc.AbstractDataSourceBean - AtomikosDataSoureBean 'COTXADBMS': getConnection ( null )... - (Slf4jLogger.java:16) 
    2015-12-11 12:45:39,478 INFO [http-/0.0.0.0:8083-10] atomikos.jdbc.AbstractDataSourceBean - AtomikosDataSoureBean 'COTXADBMS': init... - (Slf4jLogger.java:16) 
    2015-12-11 12:50:39,966 WARN [Atomikos:12] icatch.imp.ActiveStateHandler - Timeout/setRollbackOnly of ACTIVE coordinator ! - (Slf4jLogger.java:12) 
    2015-12-11 13:01:06,128 WARN [http-/0.0.0.0:8083-10] datasource.pool.ConnectionPool - atomikos connection pool 'COTXADBMS': error creating proxy of connection an AtomikosXAPooledConnection with a SessionHandleState with 0 context(s) - (Slf4jLogger.java:24) 
    com.atomikos.datasource.pool.CreateConnectionException: Error executing testQuery
        at com.atomikos.jdbc.AtomikosXAPooledConnection.testUnderlyingConnection(AtomikosXAPooledConnection.java:128)
        at com.atomikos.datasource.pool.AbstractXPooledConnection.createConnectionProxy(AbstractXPooledConnection.java:71)
        at com.atomikos.datasource.pool.ConnectionPool.retrieveFirstAvailableConnection(ConnectionPool.java:209)
        at com.atomikos.datasource.pool.ConnectionPool.retrieveFirstAvailableConnectionAndGrowPoolIfNecessary(ConnectionPool.java:173)
        at com.atomikos.datasource.pool.ConnectionPool.findOrWaitForAnAvailableConnection(ConnectionPool.java:160)
        at com.atomikos.datasource.pool.ConnectionPool.borrowConnection(ConnectionPool.java:151)
        at com.atomikos.jdbc.AbstractDataSourceBean.getConnection(AbstractDataSourceBean.java:342)
        at com.atomikos.jdbc.AbstractDataSourceBean.getConnection(AbstractDataSourceBean.java:394)
        at org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider.getConnection(InjectedDataSourceConnectionProvider.java:70)
    ...
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:222)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:814)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:737)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:754)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
        at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:164)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:121)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
        at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:202)
        at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:180)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:420)
        at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50)
        at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50)
        at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926)
        at java.lang.Thread.run(Thread.java:745)
    Caused by: java.sql.SQLRecoverableException: Erro de ES: Connection timed out
        at oracle.jdbc.driver.T4CStatement.executeForDescribe(T4CStatement.java:886)
        at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1167)
        at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1289)
        at oracle.jdbc.driver.OracleStatement.executeInternal(OracleStatement.java:1909)
        at oracle.jdbc.driver.OracleStatement.execute(OracleStatement.java:1871)
        at oracle.jdbc.driver.OracleStatementWrapper.execute(OracleStatementWrapper.java:318)
        at com.atomikos.jdbc.AtomikosXAPooledConnection.testUnderlyingConnection(AtomikosXAPooledConnection.java:124)
        ... 108 more
    Caused by: java.net.SocketException: Connection timed out
        ...

You can think, "Database can't connect man!", we have two datasources connecting to same instance, same base, same user, the first datasource is not XA and is working correctly, the second can't connect after some time.

Detail: Even you didn't use JBOSS the datasource freezes after some time. These environments are JBOSS AS 6.2

Somehow on environment B, there is some little difference in JBOSS and XA that causes this error, after JBOSS restarts, the problem stops. After some time running, the XA connection stops working, even if you send transactions or not. Now is the question, what can it blocking the thread and the connection in JBOSS?

If you wanna see what code line of Atomikos freezes check it here: http://grepcode.com/file/repo1.maven.org/maven2/com.atomikos/transactions-jdbc/3.6.4/com/atomikos/jdbc/AtomikosXAPooledConnection.java#124

Encumbrance