Error invoking timeout for timer - could not obtain lock within 5MINUTES at EJB 3 timerservice

Question

I have an application running on jboss 6.1 that defines a lot of dinamyc timers at the startup (e.g doSomething every minute) based on informations already persisted on the database. The timers are created programmatically based on these informations:

TimerConfig timerConfig = new TimerConfig();
timerConfig.setInfo(info);
timerConfig.setPersistent(false);
Timer timer = timerService.createCalendarTimer(scheduleExpression,
            timerConfig);

Today i found that the "every minute" timer created was not working anymore. Checking yesterday log, i found this strange error (full strack trace below)

Error invoking timeout for timer: [id=32b0902e-d1ee-4090-9938-98349a20340d timedObjectId=jboss.j2ee:ear=myear.ear,jar=myjar.jar,name=AppScheduler,service=EJB3 auto-timer?:false persistent?:false 
timerService=org.jboss.ejb3.timerservice.mk2.TimerServiceImpl@4036a060 initialExpiration=Thu Jan 17 00:00:00 GMT-02:00 2013 intervalDuration(in milli sec)=0 nextExpiration=Sun Jan 20 06:06:00 GMT-02:00 2013 timerState=IN_TIMEOUT: 
javax.ejb.ConcurrentAccessTimeoutException: EJB 3.1 PFD2 4.8.5.5.1 
concurrent access timeout on [advisedMethod=public void my.app.AppScheduler.process(javax.ejb.Timer), unadvisedMethod=public void my.app.AppScheduler.process(javax.ejb.Timer), metadata=null, targetObject=my.app.AppScheduler@97672ba, arguments=[Ljava.lang.Object;@3f661630]
- could not obtain lock within 5MINUTES
    at org.jboss.ejb3.concurrency.aop.interceptor.ContainerManagedConcurrencyInterceptor.invoke(ContainerManagedConcurrencyInterceptor.java:176) [:1.0.0-alpha-4]
    at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86) [jboss-aop.jar:2.2.2.GA]
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.2.GA]
    at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47) [:1.7.21]
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.2.GA]
    at org.jboss.ejb3.tx.StatelessBMTInterceptor.handleInvocation(StatelessBMTInterceptor.java:100) [:1.0.4]
    at org.jboss.ejb3.tx.BMTInterceptor.invoke(BMTInterceptor.java:57) [:1.0.4]
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.2.GA]
    at org.jboss.ejb3.tx2.aop.NoOpInterceptor.invoke(NoOpInterceptor.java:45) [:0.0.2]
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.2.GA]
    at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76) [:1.0.0.GA]
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.2.GA]
    at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:41) [:1.7.21]
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.2.GA]
    at org.jboss.ejb3.BlockContainerShutdownInterceptor.invoke(BlockContainerShutdownInterceptor.java:67) [:1.7.21]
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.2.GA]
    at org.jboss.ejb3.core.context.CurrentInvocationContextInterceptor.invoke(CurrentInvocationContextInterceptor.java:47) [:1.7.21]
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.2.GA]
    at org.jboss.aspects.currentinvocation.CurrentInvocationInterceptor.invoke(CurrentInvocationInterceptor.java:67) [:1.0.1]
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.2.GA]
    at org.jboss.ejb3.interceptor.EJB3TCCLInterceptor.invoke(EJB3TCCLInterceptor.java:86) [:1.7.21]
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.2.GA]
    at org.jboss.ejb3.singleton.aop.impl.AOPBasedInterceptorRegistry.intercept(AOPBasedInterceptorRegistry.java:111) [:1.0.2]
    at org.jboss.ejb3.singleton.impl.container.SingletonContainer.invoke(SingletonContainer.java:206) [:1.0.2]
    at org.jboss.ejb3.singleton.aop.impl.AOPBasedSingletonContainer.callTimeout(AOPBasedSingletonContainer.java:888) [:1.0.2]
    at org.jboss.ejb3.singleton.aop.impl.AOPBasedSingletonContainer.callTimeout(AOPBasedSingletonContainer.java:837) [:1.0.2]
    at org.jboss.ejb3.timerservice.mk2.task.CalendarTimerTask.callTimeout(CalendarTimerTask.java:84) [:1.0.0-alpha-13]
    at org.jboss.ejb3.timerservice.mk2.task.TimerTask.run(TimerTask.java:127) [:1.0.0-alpha-13]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) [:1.6.0_24]
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [:1.6.0_24]
    at java.util.concurrent.FutureTask.run(FutureTask.java:138) [:1.6.0_24]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98) [:1.6.0_24]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206) [:1.6.0_24]
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_24]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_24]
    at java.lang.Thread.run(Thread.java:662) [:1.6.0_24]

The main problem is not the error on a single execution, but that after this problem the timer stops working and only starts running again if the jboss is restarted. Anyone knows a way to prevent this behaviour? The exception mentions a 5 minute timeout but i dont see where to change this.

Thanks in advance.

Wether

Answer 1

Here's what the specification says about this exception

4.8.5.5.1 Concurrent Access Timeouts

A concurrent access attempt that can not immediately acquire the appropriate lock is blocked until it can make forward progress. @AccessTimeout is used to specify the amount of time the access attempt should be blocked before timing out. Access timeouts only apply to methods eligible for concurrency locks on a Singleton bean with container managed concurrency. If an access attempt times out, the container throws a javax.ejb.ConcurrentAccessTimeoutException to the client. @AccessTimeout can be specified on a business method or on a bean class (or super-class). An @AccessTimeout specified on a class applies the access timeout to all business methods of that class. If @AccessTimeout is specified on both a class and on a business method of that class, the method-level annotation takes precedence. An @AccessTimeout value of -1 indicates that the client request will block indefinitely until forward progress can be made. An @AccessTimeout value of 0 indicates that concurrent access is not allowed. Access attempts on methods with a timeout value of 0 result in a javax.ejb.ConcurrentAccessException

So, I just included access timeout to solve my issue (that 5 minutes default time is vendor specific).

@Timeout
@AccessTimeout(value = 20, unit = TimeUnit.MINUTES)
public void process(Timer timer) {
 //code here
}

Wether

Answer 2

In my case the problem was related to the last suddenly shutdown. I deleted the timers under wildfly/standalone/data/timer-service-data and deployed again.

Unquiet