stop Spring Scheduled execution if it hangs after some fixed time

Question

I have used Spring Framework's Scheduled to schedule my job to run at every 5 mins using cron. But sometime my job waits infinitely for an external resource and I can't put timeout there. I can't use fixedDelay as previous process sometime goes in wait infinitely mode and I have to refresh data at every 5 mins.

So I was looking any option in Spring Framework's Scheduled to stop that process/thread after a fixed-time either it run successfully or not.

I have found below setting which initialized ThreadPoolExecutor with 120 seconds for keepAliveTime which I put in @Configuration class. Can anybody tell me will this work as I expected.

@Bean(destroyMethod="shutdown")
public Executor taskExecutor() {
    int coreThreads = 8;
    int maxThreads = 20;
    final ThreadPoolExecutor threadPoolExecutor = new ThreadPoolExecutor(
            coreThreads, maxThreads, 120L, 
            TimeUnit.SECONDS, new LinkedBlockingQueue<Runnable>()
    );
    threadPoolExecutor.allowCoreThreadTimeOut(true);

    return threadPoolExecutor;
}

Susurrate

Your code won't work as you expect it should. Describe more details about your randomly hanging job. – Tunis

Answer 1

I'm not sure this will work as expected. Indeed the keepAlive is for IDLE thread and I don't know if your thread waiting for resources is in IDLE. Furthermore it's only when the number of threads is greater than the core so you can't really know when it happen unless you monitor the threadpool.

keepAliveTime - when the number of threads is greater than the core, this is the maximum time that excess idle threads will wait for new tasks before terminating.

What you can do is the following:

public class MyTask {

    private final long timeout;

    public MyTask(long timeout) {
        this.timeout = timeout;
    }

    @Scheduled(cron = "")
    public void cronTask() {
        Future<Object> result = doSomething();
        result.get(timeout, TimeUnit.MILLISECONDS);
    }

    @Async
    Future<Object> doSomething() {
        //what i should do
        //get ressources etc...
    }
}

Don't forget to add @EnableAsync

It's also possible to do the same without @Async by implementing a Callable.

Edit: Keep in mind that it will wait until timeout but the thread running the task won't be interrupted. You will need to call Future.cancel when TimeoutException occurs. And in the task check for isInterrupted() to stop the processing. If you are calling an api be sure that isInterrupted() is checked.

Realgar

This won't help. Topic starter have some problem inside his task code leading to randomly block work-thread forever. I wrote about it in my answer. Without resolving it - all threads from pool (no matter it's Scheduled or Async Pool) will be exhausted sooner or later (and don't forget about memory consumption for stack by every blocked thread). – Tunis

You are probably right but we don't have any info. And OP ask how he can timeout a task what ever this task do. – Realgar

Answer 2

allowCoreThreadTimeOut and timeout setting doesn't help cause it just allow work thread to be ended after some time without work (See javadocs)

You say your job waits infinitely for an external resource. I'am sure it's because you (or some third-party library you using) use sockets with time out infinite-by-default. Also keep in mind what jvm ignores Thread.interrupt() when it blocked on socket.connect/read.

So find out witch socket library used in your task (and how exactly it used) and change it's default timeout settings.

As example: there is RestTemplate widely used inside Spring (in rest client, in spring social, in spring security OAuth and so on). And there is ClientHttpRequestFactory implementation to create RestTemplate instances. By default, spring use SimpleClientHttpRequestFactory which use JDK sockets. And by default all it's timeouts are infinite.

So find out where exactly you freeze, read it's docs and configure it properly.

P.S. If you don't have enough time and "feeling lucky" try to run your app with setting jvm properties sun.net.client.defaultConnectTimeout and sun.net.client.defaultReadTimeout to some reasonable values (See docs for more details)

Tunis

If you are using java mail, following 2 configurations can be set. mail.pop3.connectiontimeout mail.pop3.timeout – Vereen

Good comments, really help. – Sepoy

Answer 3

The keepAliveTime is just for cleaning out worker threads that hasn't been needed for a while - it doesn't have any impact on the execution time of the tasks submitted to the executor.

If whatever is taking time respects interrupts you can start a new thread and join it with a timeout, interrupting it if it doesn't complete in time.

public class SomeService {

    @Scheduled(fixedRate = 5 * 60 * 1000)
    public void doSomething() throws InterruptedException {
        Thread taskThread = new TaskThread();
        taskThread.start();
        taskThread.join(120 * 000);
        if(taskThread.isAlive()) {
            // We timed out
            taskThread.interrupt();
        }
    }

    private class TaskThread extends Thread {

        public void run() {
            // Do the actual work here
        }
    }
}

Waylay

This won't help. I wrote about it in my answer. Thread blocked on socket connect/read ignores interruption. Only socket.close can help. But all this happens deep inside third-party code so TS cant close socket as hi does't "own" it. – Tunis

You're basing your entire criticism on a guess you made about what's causing the hangup. – Waylay