How to stream response body with apache HttpClient
Asked Answered
H

2

10

There's an api I need to perform octet-streaming from which does not have a length. It is just a stream of real time data. The issue that I'm having is that when I make my request, it seems to try to wait out for the end of the content before reading information into the inputstream, however it's not seeing the end of the content and timingout with NoHttpResponse exception. Below is a simplified version of my code:

private static HttpPost getPostRequest() {
    // Build uri
    URI uri = new URIBuilder()
            .setScheme("https")
            .setHost(entity.getStreamUrl())
            .setPath("/")
            .build();

    // Create http http
    HttpPost httpPost = new HttpPost(uri);

    String nvpsStr = "";
    Object myArray[] = nvps.toArray();
    for(int i = 0; i < myArray.length; i ++) {
        nvpsStr += myArray[i].toString();
        if(i < myArray.length - 1) {
            nvpsStr += "&";
        }
    }

    // Build http payload
    String request = nvpsStr + scv + streamRequest + "\n\n";
    // Attach http data
    httpPost.setEntity(new StringEntity(URLEncoder.encode(request,"UTF-8")));

    return httpPost;
}

// Where client is simply
// private static final CloseableHttpClient client = HttpClients.createDefault();
private static runPostRequest (HttpPost request) {
    CloseableHttpResponse response = client.execute(request);
    try {
        HttpEntity ent = response.getEntity();
        InputStream is = ent.getContent();
        DataInputStream dis = new DataInputStream(is);
        // Only stream the first 200 bytes
        for(int i = 0; i < 200; i++) {
            System.out.println(( (char)dis.readByte()));
        }

    } finally {
        response.close();
    }
}
Hesitate answered 9/12, 2014 at 3:58 Comment(5)
This works fine for me. Maybe show us your server side.Hanan
I have no control of server side code. This is just a 3rd party api that I'm using to stream stock quotes.Hesitate
Can you give us an example with it? I tried with a simple servlet that would stream a few bytes every few seconds and I had no problems receiving content.Hanan
Unfortunately the api I use had me sign an NDA. I'll probably check with their staff on this issue then. Thanks for verifying the validity of this code.Hesitate
possible duplicate of Apache HTTPClient Streaming HTTP POST Request?Tubby
U
8

EDIT 2

So, if you're not comfortable with threads/runnables/Handlers and not comfortable with android AsyncTask, I would just go straight to HttpUrlConnection (drop the entire excercise with Apache HttpClient because, basically Google says that HttpUrlConnection will support streamed response and it does work!)

It may not be as easy instrumenting all the details like dumping headers. But with a normal streamed response Object, I think that it should just work.... see edit 3 for HttpsUrlConnection code sample

EndEdit2

Not clear from the question what 'stream' protocol is being used (progressive download or HTTP streaming) OR how you are actually managing the streamed response on your client.

Recommended to dump the headers from the connection to see exactly what the client and server are agreeing on??

I'm assuming that you are OFF the UI thread (either in AsyncTask or in the callback portion of a Handler); if that's not accurate you may have to refactor a little bit.

Assuming HTTP stream in use with Apache HttpClient 4.3.5+

If there is no length in the headers of the response, then you are doing a 'chunked' response on HTTP 1.1 where you have to read a buffer until you get a 'last-chunk' or decide to CLOSE either the stream or the Connection:

The server just starts sending (streaming) and the client should handle the 'input-stream' that it gets from the HTTP response by employing a buffer as per the detailed Apache notes on producing entity content.

I don't remember offhand if socket timeout of 30 seconds will pre-empt an active stream? Remember in Apache, separate settings exist in the builder for socket timeout, and read timeout. Don't want socket to close on you and don't want to timeout waiting on a readable stream's available bytes while server is providing the response.

Anyway, the client-side handler just needs to be aware of how the stream ends by inspection of what's read into the buffer...

If the protocol in place is "continue" & "chunked" then the response handler on the client should be in a stream handler loop until it sees the LAST-CHUNK from the http spec.

 response.getEntity().getContent() 

should give you the reference you need to process the response's stream until 'last-chunk'...

I think u should read here on how to consume a buffered entity where more than a single read is going to be require to wind up at the 'last-chunk' in the response. It's another reason why HttpURLConnection may be easier...

Do a loop that handles buffered reads until END signaled by the bytes matching 'last-chunk'.

Then close either the stream or connection as per the detailed Apache notes on consuming entities and reusable Connections.

EDIT code for streamed response in apache HttpClient

In a 'handler's callback or in asyncTask

 request.execute();
...

 processStreamingEntity(response.getEntity());
 response.close();

//implement your own wrapper as mentioned in apache docs

    private void processStreamingEntity(HttpEntity entity) throws IOException {
        InputStreamHttpEntityHC4 bufHttpEntity = new InputStreamHttpEntityHC4(entity);
        while not bufHttpEntity.LAST_CHUNK {
            handleResponse(bufHttpEntity.readLine())
}

EDIT 3

HttpURLConnection version if you go that way. ( uses a MessageHandler but you could consume the bytes in place as this is from a streaming speach example and the words from text are being sent back to UI here)

private void openHttpsConnection(String urlStr, Handler mhandler) throws IOException {
    HttpsURLConnection httpConn = null;
    String line = null;
    try {
        URL url = new URL(urlStr);
        URLConnection urlConn = url.openConnection();               
        if (!(urlConn instanceof HttpsURLConnection)) {
            throw new IOException ("URL is not an Https URL");
        }               
        httpConn = (HttpsURLConnection)urlConn;
        httpConn.setAllowUserInteraction(false);
        httpConn.setInstanceFollowRedirects(true);
        httpConn.setRequestMethod("GET");
        httpConn.setReadTimeout(50 * 1000);
        BufferedReader is =
                new BufferedReader(new InputStreamReader(httpConn.getInputStream()));                   
        
        while ((line = is.readLine( )) != null) {

                Message msg = Message.obtain();
                msg.what=1;  
                msg.obj=line;                       
                mhandler.sendMessage(msg);
            
        }               
    } catch (MalformedURLException e) {
        e.printStackTrace();
    } catch( SocketTimeoutException e){
        e.printStackTrace();
    } catch (IOException e) {

        e.printStackTrace();
        Message msg = Message.obtain();
            msg.what=2;
            BufferedInputStream in = new BufferedInputStream(httpConn.getErrorStream());
    
            line =new String(readStream(in));
            msg.obj=line;
            mhandler.sendMessage(msg);
          
    }
    finally {httpConn.disconnect();}

}
Unorganized answered 13/12, 2014 at 15:11 Comment(9)
It's supposed to be http streaming which I've never dealt with until recently. I was using URL.openStream() to receive my data because it didn't close out my request prematurely... unlike the httpclient connection.Hesitate
hmmm. from your code looks like an apache httpclient. IMO with 4.3.5 + if you set the headers for 'continue' and for chunked, i thought that the pseudocode in my edit would consume the response.entity.Unorganized
I haven't tried it yet, I'll check it out. I'm only hoping that the server supports http 1.1 as you said. I marked this the answer, because it certainly seems like what I was looking for. I'll get back to you with the results. Thank you!Hesitate
Yeah. Keep in mind that in android, with streams/entities, there may still exist some situation where httpUrlconnection will be better for handling the wrapped stream. I would experiment with apache to find right 'entity'wrapper . Then if no work , back to httpUrlConnUnorganized
Note. Curl POST with 'trace-asci ' run on very short stream will provide some good context on details.Unorganized
It seems httpget for apache blocks until the stream is finished, so doing URL.openstream was the only way to get it working. I could check apache's asnc library but it might be too much work.Hesitate
IMO - its more likely that its a really finicky process to present the wrapper and the enclosed StreamEntity derived from the response so that its read constantly as bytes are avail from the server. If you create a wrapper around the stream entity object and process it in a thread where you are reading bytes as they become available in the inputStream, dont see how that can block. Ill get around to checking it in a streaming, http api sometime. U tested it.... From formums seems that it does not get alot of use. BTW id try both GET/POSTUnorganized
github.com/lkuza2/java-speech-api/blob/master/src/com/… FWIW - look at line# 192 ... Not an android example but the httpclient from apache not really android either. anyway, that line consumes a streamed response from googles full-duplex speech api. It does not give a working android example but it shows a scanner consuming a HTTP Stream Response in a 'bytesAvail' loop like a scanner does. So if you overlay this scanner example with an apache stream entity type consumer in a proper consumer thread, I think that it may work.Unorganized
@RobertRowntree The link for line 192 has changed an is now github.com/lkuza2/java-speech-api/blob/master/src/main/java/com/…Basidium
E
0

Try RxSON: https://github.com/rxson/rxson It utilizes the JsonPath wit RxJava to read JSON streamed chunks from the response as soon as they arrive, and parse them to java objects before the reponse complete.

Example:

String serviceURL = "https://think.cs.vt.edu/corgis/datasets/json/airlines/airlines.json";
   HttpRequest req = HttpRequest.newBuilder(URI.create(serviceURL)).GET().build();
   RxSON rxson = new RxSON.Builder().build();

   String jsonPath = "$[*].Airport.Name";
   Flowable<String> airportStream = rxson.create(String.class, req, jsonPath);
   airportStream
       .doOnNext(it -> System.out.println("Received new item: " + it))
       //Just for test
       .toList()
       .blockingGet();
Exemplum answered 24/10, 2020 at 8:6 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.