HTTP Stream dropping abnormally with java jersey client

Question

For one of our project, we are consuming HTTP feed stream by using java jersey client

With the client Feed consuming is fine, but after the 10 mins of time, stream dropping abnormally; even though stream producer is up and running and producing the stream

This is what I tried;

import java.util.Date;

import javax.ws.rs.client.Client;
import javax.ws.rs.client.ClientBuilder;
import javax.ws.rs.client.WebTarget;
import javax.ws.rs.client.Invocation.Builder;
import javax.ws.rs.core.GenericType;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;

import org.glassfish.jersey.client.ChunkedInput;

public class StreamClient {
    
    private static final String BOUNDARY = "\n";

    public void makeCall() {
        System.out.println("Start Time"+ new Date()) ;
        Client client = ClientBuilder.newClient();
        BasicAuth auth = new BasicAuth();
        auth.setPassword("username");
        auth.setUserName("password");
        BasicAuthentication basicAuthentication = new BasicAuthentication(auth);
        client.register(basicAuthentication);
        WebTarget target = client.target("https://localhost:7211/stream/v1/");
        Builder request = target.request(MediaType.APPLICATION_JSON);
        Response response = request.get();
        
        final ChunkedInput<String> chunkedInput = response.readEntity(new GenericType<ChunkedInput<String>>() {
        });
        
        chunkedInput.setParser(ChunkedInput.createParser(BOUNDARY));
        String chunk;
        do {
            if((chunk = chunkedInput.read()) != null) 
                System.out.println(chunk);  
        }while (!chunkedInput.isClosed());
        System.out.println("End Time " + new Date());
    }
    
    public static void main(String[] args) {
        StreamClient client = new StreamClient();
        client.makeCall();
    }


}

If stream doesn't drop; thread has to be within the while loop; once the stream closes then thread will come out and print sysout statement

Here the question is; why chunkedinput is closed.

Here, for every payload/chunk separation on the server end it is \r\n and to make it connection active from the server, sending \n (on the client end, I have to ignore this.)

Centipede

I think there are two lines of thought here. Either the stream is closed from the server in which case you need to go check the server logs and see why the stream has been closed, or someone in the middle is terminating the stream. For example connection timout. 10 minutes is a long time, if your jetty does not timeout there might be loadbalancers or other parties involved. – Costume

@Alexandar, I didn't set any timeout config either client or server end, by default the connection timeout for jersey client is infinity. – Centipede

Verified in the server logs, didn;t find anything related to connection drop and unfortunately on the client end, don't have log4j configuration for jersey client to debug; – Centipede

Another thing you can do is checking the responsecode, e.g. it is 200 (OK) then you can assume that the processing of your request has went well, otherwise handle it somehow (read the http headers, maybe there is some info about the reasons, log to system.err, etc.), or ignore this failed requests, it is up to you. – Splatter

Try pinging the client continuously to see if there are any packets lost or intermittent connection timeouts – Curzon

I noticed your code is pointing to localhost. Even in this scenario when you are running server and client in your development host you observe the same connection drop after 10 minutes? – Nathan

i am not sure about your problem but having the connection open for such a long time may not be a good idea .you could try using pagination for your api that could limit records up to maximum of 100 . – Reproduce

Answer 1

I would recommend using wireshark to analyze the traffic exactly at the moment the connection is lost, chances are a firewall is cutting the conection.

In any case trying to debug these scenarios without dual side logging is a 50% russian rulette situation.

If you want to discard scenarios try to vary the amount of data being transfered, the parameters of the connection (using other constructor methods) and see if the timeout is consistent in time, if the timeout is depending on the amount of traffic that is usually a clue that you trigger some threshold in an intermediary device like a firewall.

Monarchist