HttpClient stuck without any exception

Question

I'm developing a long-running application that heavily uses the HttpClient from apache.

On my first test run, the application worked perfectly until it just got stuck. It wasn't stopped, it didn't throw any exception, it just sits there doing nothing.

I did a second run just now and stopped the time and it stopped after approx. 24 hours of contant running. Additionally I noticed that the internet connection of my laptop on which I had it running was terminated at the exact moment the application got stuck. I had to reboot my WLAN adapter in order to the the net running again.

The application though, didn't return to working after the connection was up again. And now, it's stuck again.

Is there any timeout controller I'm not aware of in the HttpClient? Why doesn't my application throw an exception when the connection is down?

The part that uses the client looks as follows;

public HttpUtil(ConfigUtil config) {
    this.config = config;

    client = new DefaultHttpClient();
    client.getParams().setParameter(HttpProtocolParams.USER_AGENT, this.config.getProperty("httputil.userAgent"));
}

public String getContentAsString(String url) throws ParseException, ClientProtocolException, IOException {
    return EntityUtils.toString(
            client.execute(
                    new HttpGet(url)).getEntity());
}

The application repeatedly calls httputil.getContentAsString() on the URLs it needs.

Okajima

Maybe your entity is an empty string. – Pifer

Then my method would still return that empty string, wouldn't it? Or throw an exception at the least? – Okajima

I have same problem. httpEntity.consumeContent() save me. #9505858 – Genus

did you ever solve this issue? – Perish

@Perish It was a timeout issue. When I added a timeout to the connection it was resolved IIRC (it was a long time ago...). What the underlying problem of the connection was, I don't remember. – Okajima

Answer 1

This code is now deprecated (get HttpParams, etc). A better way is:

RequestConfig defaultRequestConfig = RequestConfig.custom()
    .setCookieSpec(CookieSpecs.BEST_MATCH)
    .setExpectContinueEnabled(true)
    .setStaleConnectionCheckEnabled(true)
    .setTargetPreferredAuthSchemes(Arrays.asList(AuthSchemes.NTLM, AuthSchemes.DIGEST))
    .setProxyPreferredAuthSchemes(Arrays.asList(AuthSchemes.BASIC))
    .build();

HttpGet httpGet = new HttpGet(url);    
RequestConfig requestConfig = RequestConfig.copy(defaultRequestConfig)
    .setSocketTimeout(5000)
    .setConnectTimeout(5000)
    .setConnectionRequestTimeout(5000)
    .build();
httpGet.setConfig(requestConfig);

Halda

Answer 2

As of version 4.4, both answers by users user2393012 and Stephen C have been deprecated. I'm not sure if there is another way of doing it, but the way I do it is by using a builder paradigm, HTTPClientBuilder.

Ex.

HttpClients.custom().setConnectionTimeToLive(1, TimeUnit.MINUTES).build()

A very similar (it actually might have been OP's problem) problem to what OP mentioned also happens but is due to Apache setting the default concurrent connections to only two connections per client. The solution to this would be to increase the max connections or close them if you can.

To increase the max connections:

HttpClients.custom().setMaxConnPerRoute(100000).build()

To close connections, you can use a BasicHttpClientConnectionManager and call on the close method for

Bitch

Answer 3

I gave a similar answer in another thread (HttpClient hangs on socketRead0 with successfully executed method)

In my case, I was setting the connectionTimeout and socketTimeout on the request, but not on the connection socket used during the establishment of the SSL connection. As a result, I would sometime hang during the SSL handshake. Below is some code that sets all 3 timeouts using the v4.4 (also tested in v4.5)

// Configure the socket timeout for the connection, incl. ssl tunneling
connManager = new PoolingHttpClientConnectionManager();
connManager.setMaxTotal(200);
connManager.setDefaultMaxPerRoute(100);

SocketConfig sc = SocketConfig.custom()
    .setSoTimeout(soTimeoutMs)
    .build();

connManager.setDefaultSocketConfig(sc);

HttpClient client = HttpClients.custom()
            .setConnectionManager(connManager)
            .setConnectionManagerShared(true)
            .build();

// configure the timeouts (socket and connection) for the request
RequestConfig.Builder config = = RequestConfig.copy(RequestConfig.DEFAULT);
config.setConnectionRequestTimeout(connectionTimeoutMs);
config.setSocketTimeout(socketTimeoutMs);

HttpRequestBase req = new HttpGet(uri);
req.setConfig(config.build());

client.execute(req);

Balsam

Thanks man, your solution works, and I've been falsely thought I was rate limited. – Clachan

Answer 4

You haven't said which version of HttpClient you are using, but assuming that it is version 4, this blog article explains what to do.

DefaultHttpClient httpClient = new DefaultHttpClient();
HttpParams params = httpClient.getParams();
HttpConnectionParams.setConnectionTimeout(httpParams, connectionTimeoutMillis);
HttpConnectionParams.setSoTimeout(httpParams, socketTimeoutMillis);

Harker

I believe this is now deprecated – Halda

Answer 5

I have all the timeouts setup just fine but I found out we have on url that does http chunking but sends no results(works fine in chrome, but in http client it hangs forever even with the timeout set). Luckily I own the server and just return some garbage and it no longer hangs. This seems like a very unique bug in that http client does not handle some kind of empty chunking case well(though I could be way off)....I just know it hangs every time on that same url with empty data and that url is http chunking csv download back to our http client.

Ancohuma

I face this issue to this day. I think my bug report was closed because I was unable to explain things property and to reproduce it. But I think the best way to reproduce this bug is to disconnect the connection in the middle of a post request. I believe that will make it get stuck even with the timeouts set – Perish

Answer 6

I had the same issue, it stuck cause I didn't close DefaultHttpClient.
So this is wrong:

try{
    DefaultHttpClient httpclient = new DefaultHttpClient();
    ...
} catch (Exception e){
    e.PrintStackTrace();
}

And this is right:

try (DefaultHttpClient httpclient = new DefaultHttpClient()){
    ...
} catch (Exception e){
    e.PrintStackTrace();
}

Hope it helps someone.

Allele

Answer 7

By default, HttpClient does not timeout (which causes more problem than it helps). What you are describing could be a hardware issue, if your network adapter died, the HttpClient will hang.

Here are the parameters set to HttpParams as part of the constructor to DefaultHttpClient including

http.socket.timeout: defines the socket timeout (SO_TIMEOUT) in milliseconds, which is the timeout for waiting for data or, put differently, a maximum period inactivity between two consecutive data packets). A timeout value of zero is interpreted as an infinite timeout. This parameter expects a value of type java.lang.Integer. If this parameter is not set, read operations will not time out (infinite timeout).

That will set a timeout on the connection, so after the set timeout an exception will be thrown.

Tripping