Upload a file through an HTTP form, via MultipartEntityBuilder, with a progress bar

Question

The short version - org.apache...MultipartEntity is deprecated, and its upgrade, MultipartEntityBuilder, appears under-represented in our online forums. Let's fix that. How does one register a callback, so my (Android) app can display a progress bar as it uploads a file?

The long version - Here's the "missing dirt-simple example" of MultipartEntityBuilder:

public static void postFile(String fileName) throws Exception {
    // Based on: https://mcmap.net/q/242040/-post-multipart-request-with-android-sdk

    HttpClient client = new DefaultHttpClient();
    HttpPost post = new HttpPost(SERVER + "uploadFile");
    MultipartEntityBuilder builder = MultipartEntityBuilder.create();        
    builder.setMode(HttpMultipartMode.BROWSER_COMPATIBLE);
    builder.addPart("file", new FileBody(new File(fileName)));
    builder.addTextBody("userName", userName);
    builder.addTextBody("password", password);
    builder.addTextBody("macAddress", macAddress);
    post.setEntity(builder.build());
    HttpResponse response = client.execute(post);
    HttpEntity entity = response.getEntity();

    // response.getStatusLine();  // CONSIDER  Detect server complaints

    entity.consumeContent();
    client.getConnectionManager().shutdown(); 

}  // FIXME  Hook up a progress bar!

We need to fix that FIXME. (An added benefit would be interruptible uploads.) But (please correct me whether or not I'm wrong), all the online examples seem to fall short.

This one, http://pastebin.com/M0uNZ6SB, for example, uploads a file as a "binary/octet-stream"; not a "multipart/form-data". I require real fields.

This example, File Upload with Java (with progress bar), shows how to override the *Entity or the *Stream. So maybe I can tell the MultipartEntityBuilder to .create() an overridden entity that meters its upload progress?

So if I want to override something, and replace the built-in stream with a counting stream that sends a signal for every 1000 bytes, maybe I can extend the FileBody part, and override its getInputStream and/or writeTo.

But when I try class ProgressiveFileBody extends FileBody {...}, I get the infamous java.lang.NoClassDefFoundError.

So while I go spelunking around my .jar files, looking for the missing Def, can someone check my math, and maybe point out a simpler fix I have overlooked?

Aggi

Thanks for example. Just what I needed! – Disperse

can someone please tell me where i can get the proper library for the MultipartEntityBuilder? – Euraeurasia

Per "org.apache.http.entity.mime", it might be httpmime-4.3.jar. I list my jars in another comment... – Aggi

Philip I think I am having a similar problem with the octect stream not being real fields.. Api is rejecting it.. I am curious if you can look at my latest post and give your thoughtd – Euraeurasia

Answer 1

The winning code (in spectacular Java-Heresy(tm) style) is:

public static String postFile(String fileName, String userName, String password, String macAddress) throws Exception {

    HttpClient client = new DefaultHttpClient();
    HttpPost post = new HttpPost(SERVER + "uploadFile");
    MultipartEntityBuilder builder = MultipartEntityBuilder.create();        
    builder.setMode(HttpMultipartMode.BROWSER_COMPATIBLE);

    final File file = new File(fileName);
    FileBody fb = new FileBody(file);

    builder.addPart("file", fb);  
    builder.addTextBody("userName", userName);
    builder.addTextBody("password", password);
    builder.addTextBody("macAddress",  macAddress);
    final HttpEntity yourEntity = builder.build();

    class ProgressiveEntity implements HttpEntity {
        @Override
        public void consumeContent() throws IOException {
            yourEntity.consumeContent();                
        }
        @Override
        public InputStream getContent() throws IOException,
                IllegalStateException {
            return yourEntity.getContent();
        }
        @Override
        public Header getContentEncoding() {             
            return yourEntity.getContentEncoding();
        }
        @Override
        public long getContentLength() {
            return yourEntity.getContentLength();
        }
        @Override
        public Header getContentType() {
            return yourEntity.getContentType();
        }
        @Override
        public boolean isChunked() {             
            return yourEntity.isChunked();
        }
        @Override
        public boolean isRepeatable() {
            return yourEntity.isRepeatable();
        }
        @Override
        public boolean isStreaming() {             
            return yourEntity.isStreaming();
        } // CONSIDER put a _real_ delegator into here!

        @Override
        public void writeTo(OutputStream outstream) throws IOException {

            class ProxyOutputStream extends FilterOutputStream {
                /**
                 * @author Stephen Colebourne
                 */

                public ProxyOutputStream(OutputStream proxy) {
                    super(proxy);    
                }
                public void write(int idx) throws IOException {
                    out.write(idx);
                }
                public void write(byte[] bts) throws IOException {
                    out.write(bts);
                }
                public void write(byte[] bts, int st, int end) throws IOException {
                    out.write(bts, st, end);
                }
                public void flush() throws IOException {
                    out.flush();
                }
                public void close() throws IOException {
                    out.close();
                }
            } // CONSIDER import this class (and risk more Jar File Hell)

            class ProgressiveOutputStream extends ProxyOutputStream {
                public ProgressiveOutputStream(OutputStream proxy) {
                    super(proxy);
                }
                public void write(byte[] bts, int st, int end) throws IOException {

                    // FIXME  Put your progress bar stuff here!

                    out.write(bts, st, end);
                }
            }

            yourEntity.writeTo(new ProgressiveOutputStream(outstream));
        }

    };
    ProgressiveEntity myEntity = new ProgressiveEntity();

    post.setEntity(myEntity);
    HttpResponse response = client.execute(post);        

    return getContent(response);

} 

public static String getContent(HttpResponse response) throws IOException {
    BufferedReader rd = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));
    String body = "";
    String content = "";

    while ((body = rd.readLine()) != null) 
    {
        content += body + "\n";
    }
    return content.trim();
}

#  NOTE ADDED LATER: as this blasterpiece gets copied into various code lineages, 
#  The management reminds the peanut gallery that "Java-Heresy" crack was there
#  for a reason, and (as commented) most of that stuff can be farmed out to off-
#  the-shelf jar files and what-not. That's for the java lifers to tool up. This
#  pristine hack shall remain obviousized for education, and for use in a pinch.

#  What are the odds??

Aggi

What does builder.addTextBody("macAddress", macAddress); add to the http POST? form fields? When I POST a request with multipart/form builder, I dont see any of this data on the request on the server. – Result

Yes; it's an example of a form field in the new style. I see it come over the wire all the time. – Aggi

I think since it's a multipart/form field, it's embedded in the stream, not a regular html form field. I think I know how to get to it now, thanks. – Result

As a total Java noob, all I can do to help is report my JARs: android-support-v4.jar google-api-client-1.17.0-rc.jar google-api-client-android-1.17.0-rc.jar google-http-client-1.17.0-rc.jar google-play-services.jar httpcore-4.3.jar httpmime-4.3.jar. You could consider a fresh Question, so it appears in the leader boards... – Aggi

Hello, this code works well however I receive "?????" when I send unicode characters to a webform using it (my user is typing non latin characters in an EditText that is used to capture the data). Any idea on how to convert this text to send it as unicode? – Craggy

The internet is UTF-8 by default, but you need to ask that in a fresh question - with the "android" and "utf-8" tags. Be prepared to answer if you have examined the raw message at your server, before your website software cooks it. – Aggi

If you have a Uri instead of filename, check this answer: #3402079 – Dressage

In case anyone needs it, this where to download the necessary file from: java2s.com/Code/Jar/h/Downloadhttpmime43jar.htm – Phantasy

If you have a problem with FileBody, then you included older jars. I have just download the 4.3 version of httpclient and httpcore from repo1.maven.org/maven2/org/apache/httpcomponents and it works like a charm. – Phantasy

Hi Philip, how exactly do I get how much bytes has been written/sent to the server in order to correctly publish the progress? – Kaduna

Both my one-liner and Mike Kogan's follow-up answer show using the int st, int end arguments to write() to add them up. It's the totalSent += end line below. – Aggi

@Philip can you help on this? #30112251 – Oly

#30132725 @Aggi – Antebellum

Answer 2

Cant thank Phlip enough for that solution. Here are the final touches for adding your progressbar support. I ran it inside an AsyncTask - progress below enables you to post the progress back to a method in the AsyncTask that invokes AsyncTask.publishProgress() for your class running in the AsyncTask. The progress bar isn't exactly smooth but at least it moves. On a Samsung S4 uploading a 4MB imagefile after the preamble it was moving 4K chunks.

     class ProgressiveOutputStream extends ProxyOutputStream {
            long totalSent;
            public ProgressiveOutputStream(OutputStream proxy) {
                   super(proxy);
                   totalSent = 0;
            }

            public void write(byte[] bts, int st, int end) throws IOException {

            // FIXME  Put your progress bar stuff here!
            // end is the amount being sent this time
            // st is always zero and end=bts.length()

                 totalSent += end;
                 progress.publish((int) ((totalSent / (float) totalSize) * 100));
                 out.write(bts, st, end);
            }

Philomena

Tx back! I left out the actual progress bar for exactly this reason: For me, it's UploadTask.this.publishProgress(totalInBlocks - blocks). I didn't want to put Android-specific code into a generic answer. – Aggi

another improvement - only output that .publish if the current end is > 1 000 or enough end values have accumulated into a local variable. – Aggi

and kids, notice MK stepped the actual 'totalSent' down by a 100. Mine steps too, but via a block size enforced by this specific wire protocol we're doing. Yet in both cases, we exploit and don't abuse the 'publish()' concept with regard to display updates. The progress bar must know its current position, without calling upon the GPU for even a recalculation, so putting the if just on the other side of that function call is the most efficient way to avoid more clutter in publish's arguments. The takeaway is: Step your progress bars, algorithmically or arbitrarily. – Aggi

How to get totalSize? I am sending an image to the server. Is it a totalSize of the image or something else? – Guan

totalSize of the image - not the fun around it on the wire - because the progress bar only meters the file. In serious use, someone should add fudge-factors to both ends, so the bar gets smoother – Aggi

Answer 3

first of all: huge thanks for the original question/answer. Since HttpPost is now deprecated, I reworked it a bit though, using additional input from this article and made a micro library of it: https://github.com/licryle/HTTPPoster

It wraps the whole in an ASync task; uses the MultipartEntityBuilder & HttpURLConnection and let's you listen for callbacks.

To use:

1. Download & extract
2. In your build.gradle Module file, add the dependency:

dependencies 
{    
     compile project(':libs:HTTPPoster') 
}

3. You need a class to implement the HttpListener interface so you can listen to the callbacks. It has four callbacks in HTTPListener:

   • onStartTransfer
   • onProgress
   • onFailure
   • onResponse

4. Configure the ASyncTask & start it. Here's a quick usage:

HashMap<String, String> mArgs = new HashMap<>();
mArgs.put("lat", "40.712784");
mArgs.put("lon", "-74.005941");

ArrayList<File> aFileList = getMyImageFiles();

HttpConfiguration mConf = new HttpConfiguration(
    "http://example.org/HttpPostEndPoint",
    mArgs,
    aFileList,
    this, // If this class implements HttpListener
    null,  // Boundary for Entities - Optional
    15000  // Timeout in ms for the connection operation
    10000, // Timeout in ms for the reading operation
);

new HttpPoster().execute(mConf);

hope that can help :) Feel also free to suggest improvements! It's very recent, and I extend it as I need it.

Cheers

Rasla

someone finally cleaned up my "Java Heresy" – Aggi