Login/Register
Android: Uploading a photo in Cloudinary with progress callback in HttpURLConnection
Asked Answered
Solved javaandroidhttpurlconnectionimage-uploadingcloudinary
M

2

8

I'm trying to modify the open source library of cloudinary so that I can listen to the progress of the upload of my photo. The library class contains a MultipartUtility java class that I modified to listen to the progress of the upload.

The original code before modifications can be found on github: https://github.com/cloudinary/cloudinary_java/blob/master/cloudinary-android/src/main/java/com/cloudinary/android/MultipartUtility.java

I literally modified it to resemble the code from another cloud service CloudFS that supports progress when uploading files / images etc:

https://github.com/bitcasa/CloudFS-Android/blob/master/app/src/main/java/com/bitcasa/cloudfs/api/MultipartUpload.java

package com.cloudinary.android;

import com.cloudinary.Cloudinary;

import java.io.*;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.Map;

/**
 * This utility class provides an abstraction layer for sending multipart HTTP
 * POST requests to a web server.
 *
 * @author www.codejava.net
 * @author Cloudinary
 */
public class MultipartUtility {
    private final String boundary;
    private static final String LINE_FEED = "\r\n";
    private static final String APPLICATION_OCTET_STREAM = "application/octet-stream";
    private HttpURLConnection httpConn;
    private String charset;
    private OutputStream outputStream;
    private PrintWriter writer;
    UploadingCallback uploadingCallback;
    public final static String USER_AGENT = "CloudinaryAndroid/" + Cloudinary.VERSION;
    Long filesize;

    public void setUploadingCallback(UploadingCallback uploadingCallback) {
        this.uploadingCallback = uploadingCallback;
    }

    /**
     * This constructor initializes a new HTTP POST request with content type is
     * set to multipart/form-data
     *
     * @param requestURL
     * @param charset
     * @throws IOException
     */
    public MultipartUtility(String requestURL, String charset, String boundary, Map<String, String> headers, Long filesize) throws IOException {
        this.charset = charset;
        this.boundary = boundary;
        this.filesize = filesize;
        URL url = new URL(requestURL);
        httpConn = (HttpURLConnection) url.openConnection();
        httpConn.setDoOutput(true); // indicates POST method
        httpConn.setDoInput(true);
        httpConn.setFixedLengthStreamingMode(filesize); //added this in

        if (headers != null) {
            for (Map.Entry<String, String> header : headers.entrySet()) {
                httpConn.setRequestProperty(header.getKey(), header.getValue());
            }
        }
        httpConn.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundary);
        httpConn.setRequestProperty("User-Agent", USER_AGENT);
        outputStream = httpConn.getOutputStream();
        writer = new PrintWriter(new OutputStreamWriter(outputStream, charset), true);
    }

    public MultipartUtility(String requestURL, String charset, String boundary) throws IOException {
        this(requestURL, charset, boundary, null, 0L);
    }

    /**
     * Adds a form field to the request
     *
     * @param name  field name
     * @param value field value
     */
    public void addFormField(String name, String value) {
        writer.append("--" + boundary).append(LINE_FEED);
        writer.append("Content-Disposition: form-data; name=\"" + name + "\"").append(LINE_FEED);
        writer.append("Content-Type: text/plain; charset=" + charset).append(LINE_FEED);
        writer.append(LINE_FEED);
        writer.append(value).append(LINE_FEED);
        writer.flush();
    }

    /**
     * Adds a upload file section to the request
     *
     * @param fieldName  name attribute in {@code <input type="file" name="..." />}
     * @param uploadFile a File to be uploaded
     * @throws IOException
     */
    public void addFilePart(String fieldName, File uploadFile, String fileName) throws IOException {
        if (fileName == null) fileName = uploadFile.getName();
        FileInputStream inputStream = new FileInputStream(uploadFile);
        addFilePart(fieldName, inputStream, fileName);
    }

    public void addFilePart(String fieldName, File uploadFile) throws IOException {
        addFilePart(fieldName, uploadFile, "file");
    }

    public void addFilePart(String fieldName, InputStream inputStream, String fileName) throws IOException {
        if (fileName == null) fileName = "file";
        writer.append("--" + boundary).append(LINE_FEED);
        writer.append("Content-Disposition: form-data; name=\"" + fieldName + "\"; filename=\"" + fileName + "\"").append(LINE_FEED);
        writer.append("Content-Type: ").append(APPLICATION_OCTET_STREAM).append(LINE_FEED);
        writer.append("Content-Transfer-Encoding: binary").append(LINE_FEED);
        writer.append(LINE_FEED);
        writer.flush();

        int progress = 0;
        byte[] buffer = new byte[4096];
        int bytesRead = -1;

        while ((bytesRead = inputStream.read(buffer)) != -1) {
            outputStream.write(buffer, 0, bytesRead);
            progress += bytesRead;
/*            int percentage = ((progress / filesize.intValue()) * 100);*/
            if (uploadingCallback != null) {
                uploadingCallback.uploadListener(progress);
            }

        }
        outputStream.flush();
        writer.flush();
        uploadingCallback = null;
        inputStream.close();
        writer.append(LINE_FEED);
        writer.flush();
    }

    public void addFilePart(String fieldName, InputStream inputStream) throws IOException {
        addFilePart(fieldName, inputStream, "file");
    }

    /**
     * Completes the request and receives response from the server.
     *
     * @return a list of Strings as response in case the server returned status
     * OK, otherwise an exception is thrown.
     * @throws IOException
     */
    public HttpURLConnection execute() throws IOException {
        writer.append("--" + boundary + "--").append(LINE_FEED);
        writer.close();

        return httpConn;
    }

}

The changes I made were to add on the following to the httpURLConnection as recommended by this thread: How to implement file upload progress bar in android: httpConn.setFixedLengthStreamingMode(filesize);

I then created a simple interface to listen for the upload progress:

public interface UploadingCallback {

    void uploadListener(int progress);

}

And then I attached it while the HttpURLConnection wrote the photo:

        while ((bytesRead = inputStream.read(buffer)) != -1) {
            outputStream.write(buffer, 0, bytesRead);
            progress += bytesRead;
/*            int percentage = ((progress / filesize.intValue()) * 100);*/
            if (uploadingCallback != null) {
                uploadingCallback.uploadListener(progress);
            }

        }

The code ran but the progress of the upload doesn't seem to be measured correctly. The photo was about 365kb and the upload took about a 10th of a second (I started the upload at 17:56:55.481 and by 17:56:55.554 it was done, thats is just over 0.7 seconds). I do not believe my internet connection is that fast and expect it to take at least 5 seconds. I have a feeling it is measuring the time it took to write the photo to the buffer instead of the time it took to send it to cloudinary's servers.

How can I get it to measure the time it takes to upload the photo so that I can use the data for my progress bar?

04-24 17:56:55.481 28306-28725/com.a  upload  4096
04-24 17:56:55.486 28306-28725/com.a  upload  8192
04-24 17:56:55.486 28306-28725/com.a  upload  12288
04-24 17:56:55.486 28306-28725/com.a  upload  16384
04-24 17:56:55.487 28306-28725/com.a  upload  20480
04-24 17:56:55.487 28306-28725/com.a  upload  24576
04-24 17:56:55.487 28306-28725/com.a  upload  28672
04-24 17:56:55.487 28306-28725/com.a  upload  32768
04-24 17:56:55.491 28306-28725/com.a  upload  36864
04-24 17:56:55.492 28306-28725/com.a  upload  40960
04-24 17:56:55.493 28306-28725/com.a  upload  45056
04-24 17:56:55.493 28306-28725/com.a  upload  49152
04-24 17:56:55.493 28306-28725/com.a  upload  53248
04-24 17:56:55.493 28306-28725/com.a  upload  57344
04-24 17:56:55.494 28306-28725/com.a  upload  61440
04-24 17:56:55.494 28306-28725/com.a  upload  65536
04-24 17:56:55.494 28306-28725/com.a  upload  69632
04-24 17:56:55.494 28306-28725/com.a  upload  73728
04-24 17:56:55.494 28306-28725/com.a  upload  77824
04-24 17:56:55.495 28306-28725/com.a  upload  81920
04-24 17:56:55.495 28306-28725/com.a  upload  86016
04-24 17:56:55.495 28306-28725/com.a  upload  90112
04-24 17:56:55.495 28306-28725/com.a  upload  94208
04-24 17:56:55.495 28306-28725/com.a  upload  98304
04-24 17:56:55.495 28306-28725/com.a  upload  102400
04-24 17:56:55.495 28306-28725/com.a  upload  106496
04-24 17:56:55.496 28306-28725/com.a  upload  110592
04-24 17:56:55.496 28306-28725/com.a  upload  114688
04-24 17:56:55.496 28306-28725/com.a  upload  118784
04-24 17:56:55.497 28306-28725/com.a  upload  122880
04-24 17:56:55.498 28306-28725/com.a  upload  126976
04-24 17:56:55.498 28306-28725/com.a  upload  131072
04-24 17:56:55.498 28306-28725/com.a  upload  135168
04-24 17:56:55.498 28306-28725/com.a  upload  139264
04-24 17:56:55.499 28306-28725/com.a  upload  143360
04-24 17:56:55.506 28306-28725/com.a  upload  147456
04-24 17:56:55.510 28306-28725/com.a  upload  151552
04-24 17:56:55.510 28306-28725/com.a  upload  155648
04-24 17:56:55.514 28306-28725/com.a  upload  159744
04-24 17:56:55.515 28306-28725/com.a  upload  163840
04-24 17:56:55.517 28306-28725/com.a  upload  167936
04-24 17:56:55.517 28306-28725/com.a  upload  172032
04-24 17:56:55.518 28306-28725/com.a  upload  176128
04-24 17:56:55.518 28306-28725/com.a  upload  180224
04-24 17:56:55.518 28306-28725/com.a  upload  184320
04-24 17:56:55.519 28306-28725/com.a  upload  188416
04-24 17:56:55.519 28306-28725/com.a  upload  192512
04-24 17:56:55.519 28306-28725/com.a  upload  196608
04-24 17:56:55.519 28306-28725/com.a  upload  200704
04-24 17:56:55.520 28306-28725/com.a  upload  204800
04-24 17:56:55.525 28306-28725/com.a  upload  208896
04-24 17:56:55.526 28306-28725/com.a  upload  212992
04-24 17:56:55.527 28306-28725/com.a  upload  217088
04-24 17:56:55.530 28306-28725/com.a  upload  221184
04-24 17:56:55.530 28306-28725/com.a  upload  225280
04-24 17:56:55.530 28306-28725/com.a  upload  229376
04-24 17:56:55.530 28306-28725/com.a  upload  233472
04-24 17:56:55.530 28306-28725/com.a  upload  237568
04-24 17:56:55.531 28306-28725/com.a  upload  241664
04-24 17:56:55.532 28306-28725/com.a  upload  245760
04-24 17:56:55.532 28306-28725/com.a  upload  249856
04-24 17:56:55.532 28306-28725/com.a  upload  253952
04-24 17:56:55.533 28306-28725/com.a  upload  258048
04-24 17:56:55.533 28306-28725/com.a  upload  262144
04-24 17:56:55.535 28306-28725/com.a  upload  266240
04-24 17:56:55.540 28306-28725/com.a  upload  270336
04-24 17:56:55.540 28306-28725/com.a  upload  274432
04-24 17:56:55.541 28306-28725/com.a  upload  278528
04-24 17:56:55.541 28306-28725/com.a  upload  282624
04-24 17:56:55.543 28306-28725/com.a  upload  286720
04-24 17:56:55.545 28306-28725/com.a  upload  290816
04-24 17:56:55.545 28306-28725/com.a  upload  294912
04-24 17:56:55.547 28306-28725/com.a  upload  299008
04-24 17:56:55.547 28306-28725/com.a  upload  303104
04-24 17:56:55.547 28306-28725/com.a  upload  307200
04-24 17:56:55.547 28306-28725/com.a  upload  311296
04-24 17:56:55.547 28306-28725/com.a  upload  315392
04-24 17:56:55.548 28306-28725/com.a  upload  319488
04-24 17:56:55.548 28306-28725/com.a  upload  323584
04-24 17:56:55.548 28306-28725/com.a  upload  327680
04-24 17:56:55.548 28306-28725/com.a  upload  331776
04-24 17:56:55.549 28306-28725/com.a  upload  335872
04-24 17:56:55.549 28306-28725/com.a  upload  339968
04-24 17:56:55.549 28306-28725/com.a  upload  344064
04-24 17:56:55.550 28306-28725/com.a  upload  348160
04-24 17:56:55.550 28306-28725/com.a  upload  352256
04-24 17:56:55.551 28306-28725/com.a  upload  356352
04-24 17:56:55.551 28306-28725/com.a  upload  360448
04-24 17:56:55.552 28306-28725/com.a  upload  364544
04-24 17:56:55.554 28306-28725/com.a  upload  365790

To test this out for yourself, you will need to create a free account on cloudinary website in order to get your cloudname so you can connect your Android SDK to their services for an unsigned direct upload from android directly to their servers.

EDIT:

This is what I have tried and it still jumps from 0 - 100% in 0.7 seconds when the upload actually finishes in 7 seconds time:

    while ((bytesRead = inputStream.read(buffer)) != -1) {
        outputStream.write(buffer, 0, bytesRead);
        progress += bytesRead;
        Log.d("MultiPart", "file transferred so far: "
                + progress);
        if (uploadingCallback != null) {
            uploadingCallback.uploadListener(progress);
        }
        Log.d("Flushing", "flush the writer");
        outputStream.flush();
        writer.flush();
    }
Mitman answered 24/4, 2016 at 17:9 Comment(0)
C
3

There is a problem in the use of flush() method and the time you call update callback().

As you can see from your code every time you read part of the picture, you write it to the output buffer, but that does not mean it's sent to the server, it might be buffered, and then later on write'n to the server.

You have two options, either call outputStream.flush() after every outputStream.write(), but that will kill the performance of the upload, because you would lose the benefits of buffering.

Or you could call your updateCallback() after the outputStream.flush() at the end of your method. Because after outputStream.flush() you are certain that the data is on the server, and that progress is over.

For more info about the flush see this thread What is the purpose of flush() in Java streams?

Carbamidine answered 26/4, 2016 at 21:52 Comment(7)
Your second alternative makes very little sense because that would be also be at 100% and the whole purpose of what I'm trying to do is to get a progress indicator for the upload. I tried your first alternative and it doesn't seem to work, I have put in an edit to show you want I have tried. – Mitman
Well than you need to do something like the ug__ propose. You should call flush() once in a while :), but as I can see from your change even that is not working for you. I will need to test this and let you know, but I am 99% sure that calling flush() is the right way to go. Maybe there is problem in some other part of your code. Will need to test this, which I will do right away. – Carbamidine
Technically, calling flush on outputstream will not do anything as the docs says it does nothing: docs.oracle.com/javase/7/docs/api/java/io/… I'm casting it to DataOutputStream to see if it will make a difference now. It actually makes no difference. – Mitman
Did you try to use this: Instead of httpConn.setFixedLengthStreamingMode(filesize); use something like httpConn.setFixedLengthStreamingMode(4096); . You should not use flush once in a while in this case, because the system should flush on every 4096. – Carbamidine
Yes. It doesn't make a difference at all. To test it out yourself, try integrate the cloudinary lib into a test app - you can also follow the instructions here on how to integrate and customise the library before you integrate: #29103979 – Mitman
I might be that with httpConn.setFixedLengthStreamingMode(filesize), the operating system, just buffers that hole file in 0.7 sec, and then after that do the real upload. if you put 4096 you are telling it to send when the buffer is 4096 big. – Carbamidine
Let us continue this discussion in chat. – Carbamidine
S
1

This is a shot in the dark because I have not tested on an Android environment, however I would recommend trying the following.

Instead of using a fixed length use setChunkedStreamingMode

//httpConn.setFixedLengthStreamingMode(filesize);
httpConn.setChunkedStreamingMode(4096); // or whatever size you see fit

doing this should trigger part of the request to get sent every time you send in 4096 bytes of data and essentially flushing out the internal buffer.

You could also try manually flushing the buffer after each write, this could slow down the file upload especially if you flush to often however it would likely fix your problem. You might end up playing with buffer sizes to find a sweet spot.

while ((bytesRead = inputStream.read(buffer)) != -1) {
    outputStream.write(buffer, 0, bytesRead);
    progress += bytesRead;
    /* int percentage = ((progress / filesize.intValue()) * 100);*/
    if (uploadingCallback != null) {
        uploadingCallback.uploadListener(progress);
    }
    // trigger the stream to write its data
    outputStream.flush();
 }

With either of these changes you would likely want to let the user choose to set their own buffer size instead of passing in the total file size. EG change your constructor to the following:

MultipartUtility(String requestURL, String charset, 
                 String boundary, Map<String, String> headers, int chunkSize)
Shum answered 26/4, 2016 at 22:10 Comment(2)
I'm a bit nervous about using setChunkedStreamingMode as I read in the docs that: Old HTTP/1.0 only servers may not support this mode. I don't know what type of server cloudinary is using so it might not work. The fixedlengthstreamingmode doesn't seem to have that restriction. Anyway, I tried your flush suggestions as well and it doesn't seem to work. See edit in my question to see what I have tried. – Mitman
@Mitman The documentation states that the URLConnection implementation is designed for RFC 2616 which is the HTTP/1.1 spec, i would be very surprised if the server you are trying to connect to doesnt support that spec as it is a standard on the web these days. Im looking at the first sentence of developer.android.com/reference/java/net/HttpURLConnection.html – Shum

© 2022 - 2024 — McMap. All rights reserved.