ThreadPoolExecutor : : TaskRejectedException from Executor

Question

My application is reading messages through Jms MessageListener class and at some point of time it is throwing TaskRejectedException. I know most of you will say that the number of threads is exceeded by maxPoolSize and queue is also full.

But I observed something. The number of messages sent to the queue from which the MessageListener class is fetching messages is 10353 and my spring property for threadPoolExecutor is below :

<bean id="ticketReaderThreadPool" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor" scope="singleton" destroy-method="destroy">
    <property name="corePoolSize" value="10" />
    <property name="maxPoolSize" value="150" />
    <property name="queueCapacity" value="11000" />
</bean>

Now according to me, the maxPoolSize is more than enough to handle these many requests. So if anyone of you can give a reason apart from maxPoolSize breach then please do so.

We are facing this issue for the second time now, previously we already tried increasing the maxPoolSize but again after 15 days we are experiencing this exception for around 5000 to 8000 times a day.

Update:

This is the full stack trace of the exception:

General Exception occurred while reading from Queue/Processing the message org.springframework.core.task.TaskRejectedException: Executor [java.util.concurrent.ThreadPoolExecutor@408b9775] did not accept task: com.batman.rapid.rapidserver.sla.TicketHandler@1be5e598 at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.execute(ThreadPoolTaskExecutor.java:244) at com.batman.rapid.rapidserver.sla.JmsTicketReceiver.onMessage(JmsTicketReceiver.java:58) at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:560) at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:498) at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467) at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325) at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263) at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058) at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050) at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.RejectedExecutionException at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1774) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:768) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:656) at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.execute(ThreadPoolTaskExecutor.java:241) ... 10 more

This is the relevant code:

if (message instanceof TextMessage)
{
    textMessage = (TextMessage) message;
    ticketReaderThreadPool.execute(new TicketHandler(textMessage.getText()));
}

Below is the configuration requested :

    <!-- End of JMS Queue Support -->

    <bean id="ticketReaderThreadPool" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor" scope="singleton" destroy-method="destroy">
    <property name="corePoolSize" value="10" />
    <property name="maxPoolSize" value="150" />
    <property name="queueCapacity" value="11000" />
</bean>

    <bean id="notificationThreadPool" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor" scope="singleton" destroy-method="destroy">
            <property name="corePoolSize" value="10" />
            <property name="maxPoolSize" value="100" />
            <property name="queueCapacity" value="10000" />
    </bean>

    <bean id="notificationManager" class="com.batman.rapid.rapidserver.sla.scheduler.NotificationManager" scope="singleton">
            <property name="defaultPercent" value="80"></property>
    </bean>

    <bean id="dbUpdateThreads" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor" scope="singleton" destroy-method="destroy">
            <property name="corePoolSize" value="1" />
            <property name="maxPoolSize" value="100" />
            <property name="queueCapacity" value="10000" />
    </bean>

Pitcher

Can we see full JMS configurations? – Enquire

I didn't get you. Please provide a detail for your requirement. – Pitcher

We would like to see the relevant spring jms configuration file. – Enquire

The error means that the executor has been shutdown or that it does not have any threads available to process the task. – Porte

@KarthikPrasad Please find the configuration requested in the updates....the issue is happening when a message is received so I think only the bean configuration provided initially with the question is relevant. – Pitcher

@Porte Can you provide some more data points to what you just mentioned? – Pitcher

If you call shutdown() on a ThreadPoolExecutor and after that try to submit a job using execute(), a TaskRejectedException will be thrown. Also, if all threads of the ThreadPoolExecutor are occupied, this exception will be thrown. What do the jobs look like that you submit? Do they run for a long time? Do they never return from the run() method? – Porte

Jobs are taking merely milliseconds to complete the execution. And as you can see, my queueCapacity configuration is more than enough to handle the number of messages my application is reading. So I don't think there is any scope of all threads being in occupied state. – Pitcher

You might be interested in : #8183705 – Freshwater

Answer 1

Not sure what is the exact cause. However I strongly feel that, the max message count might be going beyond the maxqueueCapacity you had set. However the vary approach you are using deviates from general usage of JMS. Generally in DefaultMessageListener we configure the number of Max-consumers to process the in parallel for async processing. based on the server resources we configure the max consumer that server can handle

However in your case from DefaultMessageListener the message is read and you spawn new thread and execute your business logic in new thread. since message listener read is faster than business logic, task gets accumulated in the thread task queue.

I would suggest to re-approach your current implementation. as it doesn't support transactions. I mean when server crashes all pending message processing in tasks would be terminated/lost and since you would set auto-ack, the message would have been removed from queue. Other issue like to set manually limit of thread queue etc.

Finally coming back to your issue, you can do a quick check if there are any locks in the 150 threads you had set

Enquire