How to disable buffering in Java HttpURLConnection for multi-part form post?

Question

(This is for a signed applet and I have decided against HTTPClient to keep my jar very small)

I am using HttpURLConnection to successfully upload a file from the user to a server using multi-part form post.

The problem is that HttpURLConnection is caching the data -- before sending it. So when I am reading from the file and writing to Outputstream, it is merely buffering the data -- and therefore my progress bar, that shows the upload status , is completely wrong. Howevere please note that the form post code works and the file does get uploaded correctly with return code of 200.

So how do I ensure that HttpURLConnection does not cache the data that I am sending to the server ?

Here is my source:

public UploadResponse send(String formPostUrlStr,String fileFieldName,File targetFile, Map<String, String> valuesMap, UploadStatusListener uploadStatusListener) throws Exception{


    String sendStr=getBoundaryMessage(Boundary,  valuesMap, fileFieldName, targetFile.getName(), valuesMap.get("content-type") );//"image/png") ;

    System.out.println(" multi-part start \n "+ sendStr+ " multi-part end \n");

    String lenstr=Long.toString((long)(sendStr.length()*2)+ targetFile.length());
    System.out.println("Content-Length"+ lenstr);
                        //Content-Length

    URL url= new URL(formPostUrlStr);
    long startTime= System.currentTimeMillis();
    HttpURLConnection s3Connection = (HttpURLConnection) url.openConnection();

    System.out.println("opened url to "+ formPostUrlStr +", connection ok ..");
    s3Connection.setRequestProperty("Content-Type", "multipart/form-data; boundary="
            + Boundary);
    s3Connection.setRequestProperty("content-length",  lenstr);




    s3Connection.setDoOutput(true);
    s3Connection.setDoInput(true);
    s3Connection.setUseCaches(false);
     s3Connection.setInstanceFollowRedirects(true);
    s3Connection.setAllowUserInteraction(true);
    s3Connection.setRequestProperty("User-Agent", "Mozilla/4.5");


    if (uploadStatusListener != null) {
        uploadStatusListener.statusUpdate(targetFile.length(), 0);
    }


    String debugStr= s3Connection.toString();
    System.out.println("conmnection "+ debugStr);


    DataOutputStream httpOut = new DataOutputStream(s3Connection.getOutputStream());

    System.out.println("opened DataOutputStream ok ..");




    httpOut.write(sendStr.getBytes());


    //httpOut.flush();

  System.out.println("httpOut.flush 1 ok ..");
    FileInputStream uploadFileReader = new FileInputStream(targetFile);
    long totalBytes = uploadFileReader.available();
    if (uploadStatusListener != null) {
        uploadStatusListener.statusUpdate(totalBytes, 0);
    }
    System.out.println(" uploading file with size  "+ uploadFileReader.available());
    int bufSize = 102400;
    long availableBytesToRead;
    long totalSent = 0;
    while ((availableBytesToRead = uploadFileReader.available()) > 0) {
        byte[] bufferBytesRead;
        bufferBytesRead = availableBytesToRead >= bufSize ? new byte[bufSize]
                : new byte[(int)availableBytesToRead];
        int count = uploadFileReader.read(bufferBytesRead);
        try{
            httpOut.write(bufferBytesRead);
            totalSent += ((long) count);
            System.out.println(" wrote bytes = "+count+ ", total sent = "+ totalSent +", pendingSize"+ (availableBytesToRead-count) );
        }
        catch(IOException ioe){
            System.out.println(" io exceotion e"+ ioe.getMessage());
            throw ioe;
        }

        //httpOut.flush();
        if (uploadStatusListener != null) {
            uploadStatusListener.statusUpdate(totalBytes, totalSent);
        }

    }
    // FILE DATA END 
    httpOut.write(("--" + Boundary + "--\r\n").getBytes());



    // form end     
    httpOut.write(("--" + Boundary + "--\r\n").getBytes());

    httpOut.flush();
    httpOut.close();

    long endTime= System.currentTimeMillis();

    System.out.println("Completed Writing Data to S3 Connection in "+ (endTime-startTime)+"ms.,now waiting for rsponse code ");
    int code=s3Connection.getResponseCode();
    long endTime2= System.currentTimeMillis();
    System.out.println("Completed Sendind Data to S3 in "+ (endTime2-startTime)+ "ms., rsponse code time "+ (endTime2-endTime)+"ms. ");


    UploadResponse uploadResponse = new UploadResponse();

    uploadResponse.setCode(code);

    System.out.println(" response code : " + code);

    StringBuilder response = new StringBuilder();
    byte[] respBuffer = new byte[4096];


    if (code > 300) {
        if (code == 404) {
            throw new Exception("Error 404");
        }
        BufferedReader err = new BufferedReader(
                new InputStreamReader(s3Connection.getErrorStream()));
        String ret;
        StringBuffer buff = new StringBuffer();
        while ((ret = err.readLine()) != null) {
            buff.append(ret);
        }
        uploadResponse.setMessage(buff.toString());
        System.out.println(" error :"+ buff.toString());
        err.close();


    } else {
        BufferedReader inp = new BufferedReader(
                new InputStreamReader(s3Connection.getInputStream()));
        StringBuffer buff = new StringBuffer();
        String ret;
        while ((ret = inp.readLine()) != null) {
            buff.append(ret);
        }
        inp.close();
        uploadResponse.setMessage(buff.toString());
        if(buff.toString().contains("fail"))
            throw new Exception("Upload failed");
    }        

    System.out.println(response.toString());

    return uploadResponse;        

}

}

Cassiterite

Answer 1

I have the same problem. I didn't find any other solution than writing my HTTP request on a raw Socket. Did you find a better workaround ?

EDIT : I just did : we just have to use obj.setFixedLengthStreamingMode(12345) on the HttpURLConnection object obtained from url.openConnection(), where 12345 is the length of POST request body.

Allembracing

Answer 2

As a complementation for the answer that @Antares gave, there is another method setChunkedStreamingMode that is used when you don't know the content size in advance. So when you do a POST request, call that method on the connection:

HttpURLConnection connection = ...
connection.setRequestMethod("POST");
connection.setDoOutput(true);
connection.setChunkedStreamingMode(0);
connection.connect();
... connection.getOutputStream();

This will avoid the OutputStream to buffer the entire content before start to send.

Isolation