Apache HttpClient timeout
Asked Answered
S

7

59

Is there a way to specify a timeout for the whole execution of HttpClient?

I have tried the following:

httpClient.getParams().setParameter("http.socket.timeout", timeout * 1000);
httpClient.getParams().setParameter("http.connection.timeout", timeout * 1000);
httpClient.getParams().setParameter("http.connection-manager.timeout", new Long(timeout * 1000));
httpClient.getParams().setParameter("http.protocol.head-body-timeout", timeout * 1000);

It actually works fine, except if a remote host sends back data - even at one byte/second - it will continue to read forever! But I want to interrupt the connection in 10 seconds max, whether or not the host responds.

Sd answered 20/7, 2011 at 15:3 Comment(2)
See https://mcmap.net/q/331008/-httpclient-timeoutLanky
If it is async, there is a timeout. But it is sync, so no way as per in the HttpClient way.Scruple
S
39

There is currently no way to set a maximum request duration of that sort: basically you want to say I don't care whether or not any specific request stage times out, but the entire request must not last longer than 15 seconds (for example).

Your best bet would be to run a separate timer, and when it expires fetch the connection manager used by the HttpClient instance and shutdown the connection, which should terminate the link. Let me know if that works for you.

Spreader answered 20/7, 2011 at 15:24 Comment(3)
Yep, probably will have to do that: the Timer should run in a thread by itself.Spreader
Is this worth a feature request? It seems like a common use caseRuthi
Here you find an implementation of such solution, see baeldung.com/httpclient-timeout#hard_timeoutBaudoin
E
63

For a newer version of httpclient (e.g. http components 4.3 - https://hc.apache.org/httpcomponents-client-4.3.x/index.html):

int CONNECTION_TIMEOUT_MS = timeoutSeconds * 1000; // Timeout in millis.
RequestConfig requestConfig = RequestConfig.custom()
    .setConnectionRequestTimeout(CONNECTION_TIMEOUT_MS)
    .setConnectTimeout(CONNECTION_TIMEOUT_MS)
    .setSocketTimeout(CONNECTION_TIMEOUT_MS)
    .build();

HttpPost httpPost = new HttpPost(URL);
httpPost.setConfig(requestConfig);
Errick answered 12/4, 2015 at 17:17 Comment(4)
Added the units, I hate it when it just says setTimeout, should be setTimeoutMs, it is indeed milliseconds.Granivorous
Wouldn't this setup allow a total request lifecycle > CONNECTION_TIMEOUT_MS? You've specified the same timeout at multiple stages of the request.Nitrosyl
I don't think this is the correct answer to the original question. The documentation for the connection request timeout states. Returns the timeout in milliseconds used when requesting a connection from the connection manager this is NOT the total time executing the requestion just to get the connection from the connection manager. Therefore if the server was returning 1 btye/s , as the OP asks this could easily exceed the connection request timeout.Paramorph
Agree, this is not a solution for the given problem. If request keeps reading packets from the socket the request will never timeout. Question was about hard timeout.Borneol
S
39

There is currently no way to set a maximum request duration of that sort: basically you want to say I don't care whether or not any specific request stage times out, but the entire request must not last longer than 15 seconds (for example).

Your best bet would be to run a separate timer, and when it expires fetch the connection manager used by the HttpClient instance and shutdown the connection, which should terminate the link. Let me know if that works for you.

Spreader answered 20/7, 2011 at 15:24 Comment(3)
Yep, probably will have to do that: the Timer should run in a thread by itself.Spreader
Is this worth a feature request? It seems like a common use caseRuthi
Here you find an implementation of such solution, see baeldung.com/httpclient-timeout#hard_timeoutBaudoin
S
11

Works fine, as proposed by Femi. Thanks!

Timer timer = new Timer();
timer.schedule(new TimerTask() {
    public void run() {
        if(getMethod != null) {
            getMethod.abort();
        }
    }
}, timeout * 1000);
Sd answered 21/7, 2011 at 9:51 Comment(5)
needs a lock, to avoid NPE. getMethod can become null between the check and the call to abort!Cannibal
@TonyBenBrahim can you show an example of how to add this lock?Risotto
@Turbo: synchronized(foo){ ... getMethod=null; } synchronized(foo){ if (getMethod!=null){ getMethod.abort(); } }Cannibal
@TonyBenBrahim thanks for that. I'm new to synchronization. The section of getMethod=null; is just an example of the method where the HttpClient is being used correct? That is, getMethod wouldn't deliberately get set to null, but might become null when that method exits, therefore it should be synchronized (on the same object) as the Timer thread.Risotto
This solution is not right for prod. Keep in in mind each timer will spin a thread. 1000/req per minute will spin 1000 thread a min. This will continue to pile up until you service die due to out of memoryBorneol
B
11

Timer is evil! Using timer or executor or any other mechanism which creates a thread/runnable object per request is a very bad idea. Please think wisely and don't do it. Otherwise you will quickly run into all kind of memory issues with more or less real environment. Imagine 1000 req/min means 1000 threads or workers / min. Poor GC. The solution I propose require only 1 watchdog thread and will save you resources time and nerves. Basically you do 3 steps.

  1. put request in cache.
  2. remove request from cache when complete.
  3. abort requests which are not complete within your limit.

your cache along with watchdog thread may look like this.

import org.apache.http.client.methods.*;
import java.util.*;
import java.util.concurrent.*;
import java.util.stream.*;

public class RequestCache {

private static final long expireInMillis = 300000;
private static final Map<HttpUriRequest, Long> cache = new ConcurrentHashMap<>();
private static final ScheduledExecutorService exe = Executors.newScheduledThreadPool(1);

static {
    // run clean up every N minutes
    exe.schedule(RequestCache::cleanup, 1, TimeUnit.MINUTES);
}

public static void put(HttpUriRequest request) {
    cache.put(request, System.currentTimeMillis()+expireInMillis);
}

public static void remove(HttpUriRequest request) {
    cache.remove(request);
}

private static void cleanup() {
    long now = System.currentTimeMillis();
    // find expired requests
    List<HttpUriRequest> expired = cache.entrySet().stream()
            .filter(e -> e.getValue() > now)
            .map(Map.Entry::getKey)
            .collect(Collectors.toList());

    // abort requests
    expired.forEach(r -> {
        if (!r.isAborted()) {
            r.abort();
        }
        cache.remove(r);
      });
    }
  }

and the following sudo code how to use cache

import org.apache.http.client.methods.*;

public class RequestSample {

public void processRequest() {
    HttpUriRequest req = null;
    try {
        req = createRequest();

        RequestCache.put(req);

        execute(req);

    } finally {
        RequestCache.remove(req);
    }
  }
}
Borneol answered 18/4, 2019 at 4:18 Comment(2)
Sorry for late comment, If we not using TIme in each call instead using your method using just 1 schedule thread how do we return call if e.abort() is called.Romans
http framework will handle this for you. As connection is aborted it will throw HTTP error. Similar to connection time-out or connection interrupted.Borneol
P
4

Building off the the other answers, my solution was to use a HttpRequestInterceptor to add the abort runnable to every request. Also i swapped out the Timer for a ScheduledExecutorService.

public class TimeoutInterceptor implements HttpRequestInterceptor {

private int requestTimeout = 1 * DateTimeConstants.MILLIS_PER_MINUTE;

private ScheduledExecutorService executorService = Executors.newSingleThreadScheduledExecutor();

public TimeoutInterceptor() {  }

public TimeoutInterceptor(final int requestTimeout) {
    this.requestTimeout = requestTimeout;
}

@Override
public void process(final HttpRequest request, final HttpContext context) throws HttpException, IOException {
    if (request instanceof AbstractExecutionAwareRequest) {
        final AbstractExecutionAwareRequest abortableRequest = (AbstractExecutionAwareRequest) request;
        setAbort(abortableRequest);
    } else if (request instanceof HttpRequestWrapper) {
        HttpRequestWrapper wrapper = (HttpRequestWrapper) request;
        this.process(wrapper.getOriginal(), context);
    }

}

/**
 * @param abortableRequest
 */
private void setAbort(final AbstractExecutionAwareRequest abortableRequest) {
    final SoftReference<AbstractExecutionAwareRequest> requestRef = new SoftReference<AbstractExecutionAwareRequest>(abortableRequest);

    executorService.schedule(new Runnable() {

        @Override
        public void run() {
            AbstractExecutionAwareRequest actual = requestRef.get();
            if (actual != null && !actual.isAborted()) {
                actual.abort();
            }
        }
    }, requestTimeout, TimeUnit.MILLISECONDS);

}

public void setRequestTimeout(final int requestTimeout) {
    this.requestTimeout = requestTimeout;
}
}
Paramorph answered 23/5, 2016 at 15:48 Comment(2)
This is working solution but you need create a runnable per request. That quickly turns into a problem on real high volume env. 1000 req per min will result in 1000 runnables per min. Runnables will be queued up. Indeed you will run into various problems causing service failure.Borneol
Actually I don't think it would cause a lot of problems. With 1000 requests a minute you would always have around 1000 runnables in the queue (with a 1 minute timeout). Since it's still only one thread, I guess that should be ok.Himes
D
2

In HttpClient 4.3 version you can use below example.. let say for 5 seconds

int timeout = 5;
RequestConfig config = RequestConfig.custom()
  .setConnectTimeout(timeout * 1000)
  .setConnectionRequestTimeout(timeout * 1000)
  .setSocketTimeout(timeout * 1000).build();
CloseableHttpClient client = 
  HttpClientBuilder.create().setDefaultRequestConfig(config).build();
HttpGet request = new HttpGet("http://localhost:8080/service"); // GET Request
response = client.execute(request);
Defect answered 19/8, 2014 at 9:40 Comment(0)
C
-5

That's also work:

HttpClient client = new DefaultHttpClient();
HttpConnectionParams.setConnectionTimeout(client.getParams(), timeout * 1000);
HttpConnectionParams.setSoTimeout(client.getParams(), timeout * 1000);
Circumfuse answered 22/8, 2013 at 14:24 Comment(1)
actually, that is exactly what he does in the question so that is obviously not working for him(nor me unfortunately either).Diella

© 2022 - 2024 — McMap. All rights reserved.