Spring Boot: gracefully shutdown by controlling termination order involving MongoClient
Asked Answered
D

1

3

I have a Spring Boot application that spawns many threads using an AsyncTaskExecutor (the number is predefined)

The threads execute an infinite loop, that reads from some queue and process objects, so I don't really have a rejection policy mechanism (like a ThreadPool that accept tasks)

The problem is that when the application gets closed, threads might (and probably) be busy with processing an item which includes operations to Mongo using MongoTemplate.

So when the application gets closed the MongoClient is being close()'d automatically and then I'm getting some errors from Mongo, like:

java.lang.IllegalStateException: The pool is closed
    at com.mongodb.internal.connection.ConcurrentPool.get(ConcurrentPool.java:137)
    at com.mongodb.internal.connection.DefaultConnectionPool.getPooledConnection(DefaultConnectionPool.java:262)
    at com.mongodb.internal.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:103)
    at com.mongodb.internal.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:92)
    at com.mongodb.internal.connection.DefaultServer.getConnection(DefaultServer.java:85)

How can I close the application gracefully? e.g. interrupting the threads while not shutting down the MongoClient just yet?

CODE:

Bean creation:

@Bean
AsyncTaskExecutor getTaskExecutor() {
    SimpleAsyncTaskExecutor executor = new SimpleAsyncTaskExecutor();
    return executor;
}

Execution simply by:

executor.execute(runnable);
Dwyer answered 13/4, 2020 at 12:16 Comment(1)
Show how you spawn the threads. You should be able to Override isInterrupted and do a graceful disconnect from mongo.Oceanic
B
10

Don't use SimpleAsyncTaskExecutor - SimpleAsyncTaskExecutor creates a new thread for each request instead use ThreadPoolTaskExecutor and configure two properties noted below.

/**
 * Set whether to wait for scheduled tasks to complete on shutdown,
 * not interrupting running tasks and executing all tasks in the queue.
 * <p>Default is "false", shutting down immediately through interrupting
 * ongoing tasks and clearing the queue. Switch this flag to "true" if you
 * prefer fully completed tasks at the expense of a longer shutdown phase.
 * <p>Note that Spring's container shutdown continues while ongoing tasks
 * are being completed. If you want this executor to block and wait for the
 * termination of tasks before the rest of the container continues to shut
 * down - e.g. in order to keep up other resources that your tasks may need -,
 * set the {@link #setAwaitTerminationSeconds "awaitTerminationSeconds"}
 * property instead of or in addition to this property.
 * @see java.util.concurrent.ExecutorService#shutdown()
 * @see java.util.concurrent.ExecutorService#shutdownNow()
 */
public void setWaitForTasksToCompleteOnShutdown(boolean waitForJobsToCompleteOnShutdown) {
    this.waitForTasksToCompleteOnShutdown = waitForJobsToCompleteOnShutdown;
}

/**
 * Set the maximum number of seconds that this executor is supposed to block
 * on shutdown in order to wait for remaining tasks to complete their execution
 * before the rest of the container continues to shut down. This is particularly
 * useful if your remaining tasks are likely to need access to other resources
 * that are also managed by the container.
 * <p>By default, this executor won't wait for the termination of tasks at all.
 * It will either shut down immediately, interrupting ongoing tasks and clearing
 * the remaining task queue - or, if the
 * {@link #setWaitForTasksToCompleteOnShutdown "waitForTasksToCompleteOnShutdown"}
 * flag has been set to {@code true}, it will continue to fully execute all
 * ongoing tasks as well as all remaining tasks in the queue, in parallel to
 * the rest of the container shutting down.
 * <p>In either case, if you specify an await-termination period using this property,
 * this executor will wait for the given time (max) for the termination of tasks.
 * As a rule of thumb, specify a significantly higher timeout here if you set
 * "waitForTasksToCompleteOnShutdown" to {@code true} at the same time,
 * since all remaining tasks in the queue will still get executed - in contrast
 * to the default shutdown behavior where it's just about waiting for currently
 * executing tasks that aren't reacting to thread interruption.
 * @see java.util.concurrent.ExecutorService#shutdown()
 * @see java.util.concurrent.ExecutorService#awaitTermination
 */
public void setAwaitTerminationSeconds(int awaitTerminationSeconds) {
    this.awaitTerminationSeconds = awaitTerminationSeconds;
}

Relavant part

Set the maximum number of seconds that this executor is supposed to block on shutdown in order to wait for remaining tasks to complete their execution before the rest of the container continues to shut down. This is particularly useful if your remaining tasks are likely to need access to other resources that are also managed by the container.

You can configure using spring auto configuration to control the task execution properties (preferred) or programmatically with @Bean annotation

Spring boot in 2.1.0 provides auto configuration for task executors and uses for both @EnableAsync and Spring MVC Async support.

There is no task executor bean / webMvcConfigurer configuration is needed from application. So remove the one you have and should be good.

You can adjust using applicaion properties/yml file with spring.task.execution.*.

spring.task.execution.shutdown.await-termination=true
spring.task.execution.shutdown.await-termination-period=60

Full listing can be found here

More details here and here

OR

@Bean
public ThreadPoolTaskExecutor threadPoolTaskExecutor() {
    ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
    taskExecutor.setCorePoolSize(5);
    taskExecutor.setMaxPoolSize(5); 
    taskExecutor.waitForTasksToCompleteOnShutdown(true);
    taskExecutor.setAwaitTerminationSeconds(60);
    return taskExecutor;
  }
Buy answered 16/4, 2020 at 11:59 Comment(7)
I actually used ThreadPoolTaskExecutor with these parameters. The thing is that I'm still getting errors from the mongo client, requiring me to try-catch the IllegalStateException every time I'm using MongoTemplate let's say. Is there a way to get around this?Dwyer
That is odd. If you could share an example reproducing your issue (preferably github project) then I could take a deeper look and also please update your post to show what you have right now. More details are needed. Could you use @Async annotation to run your tasks ? Also how are these tasks scheduled ? You have to let spring manage the tasks for you.Buy
Some examples hereBuy
I'll try @Async as you suggestedDwyer
Sagar, can you please give a brief explanation how @Async would solve the issue?Dwyer
Once you mark the method as @Async, when the method is invoked it will run in separate thread ( thread borrowed from threadpool ) and managed by spring. So once your spring boot is shutting down it will gracefully finish the thread executing the task. You will need @EnableAsync to enable async processing.Buy
@SagarVeeram sorry but Spring does not gracefully shutdown async task. I've tested and the async thread is interrupted before it get completed.Leninist

© 2022 - 2024 — McMap. All rights reserved.