Sending files using POST with HttpURLConnection

Question

Since the Android developers recommend to use the HttpURLConnection class, I was wondering if anyone can provide me with a good example on how to send a bitmap "file" (actually an in-memory stream) via POST to an Apache HTTP server. I'm not interested in cookies or authentication or anything complicated, but I just want to have a reliable and logic implementation. All the examples that I've seen around here look more like "let's try this and maybe it works".

Right now, I have this code:

URL url;
HttpURLConnection urlConnection = null;
try {
    url = new URL("http://example.com/server.cgi");

    urlConnection = (HttpURLConnection) url.openConnection();

} catch (Exception e) {
    this.showDialog(getApplicationContext(), e.getMessage());
}
finally {
    if (urlConnection != null)
    {
        urlConnection.disconnect();
    }
}

where showDialog should just display an AlertDialog (in case of an invalid URL?).

Now, let's say that I generate a bitmap like so: Bitmap image = this.getBitmap() inside a control derived from View and I want to send it via POST. What would be the proper procedure to achieve such a thing? What classes do I need to use? Can I use HttpPost like in this example? If so, how would I construct the InputStreamEntity for my bitmap? I would find it revolting to be required to first store the bitmap in a file on the device.

---

I should also mention that I really need to send every unaltered pixel of the original bitmap to the server, so I can't convert it to JPEG.

Injunction

Refer this SO discussion – Feria

Answer 1

I have no idea why the HttpURLConnection class does not provide any means to send files without having to compose the file wrapper manually. Here's what I ended up doing, but if someone knows a better solution, please let me know.

Input data:

Bitmap bitmap = myView.getBitmap();

Static stuff:

String attachmentName = "bitmap";
String attachmentFileName = "bitmap.bmp";
String crlf = "\r\n";
String twoHyphens = "--";
String boundary =  "*****";

Setup the request:

HttpURLConnection httpUrlConnection = null;
URL url = new URL("http://example.com/server.cgi");
httpUrlConnection = (HttpURLConnection) url.openConnection();
httpUrlConnection.setUseCaches(false);
httpUrlConnection.setDoOutput(true);

httpUrlConnection.setRequestMethod("POST");
httpUrlConnection.setRequestProperty("Connection", "Keep-Alive");
httpUrlConnection.setRequestProperty("Cache-Control", "no-cache");
httpUrlConnection.setRequestProperty(
    "Content-Type", "multipart/form-data;boundary=" + this.boundary);

Start content wrapper:

DataOutputStream request = new DataOutputStream(
    httpUrlConnection.getOutputStream());

request.writeBytes(this.twoHyphens + this.boundary + this.crlf);
request.writeBytes("Content-Disposition: form-data; name=\"" +
    this.attachmentName + "\";filename=\"" + 
    this.attachmentFileName + "\"" + this.crlf);
request.writeBytes(this.crlf);

Convert Bitmap to ByteBuffer:

//I want to send only 8 bit black & white bitmaps
byte[] pixels = new byte[bitmap.getWidth() * bitmap.getHeight()];
for (int i = 0; i < bitmap.getWidth(); ++i) {
    for (int j = 0; j < bitmap.getHeight(); ++j) {
        //we're interested only in the MSB of the first byte, 
        //since the other 3 bytes are identical for B&W images
        pixels[i + j] = (byte) ((bitmap.getPixel(i, j) & 0x80) >> 7);
    }
}

request.write(pixels);

End content wrapper:

request.writeBytes(this.crlf);
request.writeBytes(this.twoHyphens + this.boundary + 
    this.twoHyphens + this.crlf);

Flush output buffer:

request.flush();
request.close();

Get response:

InputStream responseStream = new 
    BufferedInputStream(httpUrlConnection.getInputStream());

BufferedReader responseStreamReader = 
    new BufferedReader(new InputStreamReader(responseStream));

String line = "";
StringBuilder stringBuilder = new StringBuilder();

while ((line = responseStreamReader.readLine()) != null) {
    stringBuilder.append(line).append("\n");
}
responseStreamReader.close();

String response = stringBuilder.toString();

Close response stream:

responseStream.close();

Close the connection:

httpUrlConnection.disconnect();

PS: Of course I had to wrap the request in private class AsyncUploadBitmaps extends AsyncTask<Bitmap, Void, String>, in order to make the Android platform happy, because it doesn't like to have network requests on the main thread.

Injunction

Finally a perfectly explained answer for this question! Thank you! BTW, I just found this article from the Android Developers Blog (android-developers.blogspot.com/2011/09/…) where they suggest using HTTPURLConnection over the Apache HTTPClient. Cheers! – Repairer

PS: According to this, ` [error] => 3` means that "the uploaded file was only partially uploaded", so I can assume that there are some buffering issues, but I have no idea how to debug / fix something like this. – Injunction

See my comment just above. You need to add them to the url variable, like so: URL url = new URL("http://example.com/?param1=val1&param2=val2");. You can add as many as you wish (although I think there are some limits). – Injunction

Perfect answer! I suggest to take a look at this w3.org docs about forms – Natachanatal

Answer 2

I actually found a better way to send files using HttpURLConnection using MultipartEntity

private static String multipost(String urlString, MultipartEntity reqEntity) {
    try {
        URL url = new URL(urlString);
        HttpURLConnection conn = (HttpURLConnection) url.openConnection();
        conn.setReadTimeout(10000);
        conn.setConnectTimeout(15000);
        conn.setRequestMethod("POST");
        conn.setUseCaches(false);
        conn.setDoInput(true);
        conn.setDoOutput(true);

        conn.setRequestProperty("Connection", "Keep-Alive");
        conn.addRequestProperty("Content-length", reqEntity.getContentLength()+"");
        conn.addRequestProperty(reqEntity.getContentType().getName(), reqEntity.getContentType().getValue());

        OutputStream os = conn.getOutputStream();
        reqEntity.writeTo(conn.getOutputStream());
        os.close();
        conn.connect();

        if (conn.getResponseCode() == HttpURLConnection.HTTP_OK) {
            return readStream(conn.getInputStream());
        }

    } catch (Exception e) {
        Log.e(TAG, "multipart post error " + e + "(" + urlString + ")");
    }
    return null;        
}

private static String readStream(InputStream in) {
    BufferedReader reader = null;
    StringBuilder builder = new StringBuilder();
    try {
        reader = new BufferedReader(new InputStreamReader(in));
        String line = "";
        while ((line = reader.readLine()) != null) {
            builder.append(line);
        }
    } catch (IOException e) {
        e.printStackTrace();
    } finally {
        if (reader != null) {
            try {
                reader.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }
    return builder.toString();
} 

Assuming you are uploading an image with bitmap data:

    Bitmap bitmap = ...;
    String filename = "filename.png";
    ByteArrayOutputStream bos = new ByteArrayOutputStream();
    bitmap.compress(Bitmap.CompressFormat.PNG, 100, bos);
    ContentBody contentPart = new ByteArrayBody(bos.toByteArray(), filename);

    MultipartEntity reqEntity = new MultipartEntity(HttpMultipartMode.BROWSER_COMPATIBLE);
    reqEntity.addPart("picture", contentPart);
    String response = multipost("http://server.com", reqEntity);

And Voila! Your post data will contain picture field along with the filename and path on your server.

Jainism

I notice you're not calling conn.disconnect(), is it intentional? – Damaraland

@MihaiTodor You will want to compress the bitmap into a File to reduce the amount of data passed over network anyway. – Smash

@Smash Well, that is up to the developer, but my method lets people preserve the original pixels of the bitmap. In my use case (256x256 bitmaps for a demo), zipping the bitmap wasn't worth the hassle, but for a real application, it will definitely be a good practice to use some archiving mechanism. – Injunction

you should call setFixedLengthStreamingMode(reqEntity.getContentLength()) instead of setting the Content-Length header directly. This way your data isn't buffered before sending on the socket (at least on newer devices, on android 2.3 or less it seems they get buffered anyways). I fixed the buffering issue on older devices by using okhttp as the transport and that works. – Frigidarium

too bad MultipartEntity is not bundled with Android SDK – Goshen

@Goshen It is bundled with the httpmime. I use this as a Gradle dependency: org.apache.httpcomponents:httpmime:4.1.1 – Gaea

Works fine, but I had to comment conn.addRequestProperty("Content-length", reqEntity.getContentLength()+"") because it was generating a malformed request with a duplicated "Content-length:" header with value "-1". Apache was replying 400 Bad Request even before the upload was completed. Content-length is set automatically by HttpURLConnection. – Aeolipile

This its part of android now as com.android.internal.http.multipart.MultipartEntity so the gradle dependency is not needed. – Disprize

Answer 3

The solution of Jaydipsinh Zala didn't work for me, I don't know why but it seems to be close to the solution.

So merging this one with the great solution and explanation of Mihai Todor, the result is this class that currently works for me. If it helps someone:

MultipartUtility2V.java

import java.io.*;
import java.net.HttpURLConnection;
import java.net.URL;
import java.nio.file.Files;

public class MultipartUtilityV2 {
    private HttpURLConnection httpConn;
    private DataOutputStream request;
    private final String boundary =  "*****";
    private final String crlf = "\r\n";
    private final String twoHyphens = "--";

    /**
     * This constructor initializes a new HTTP POST request with content type
     * is set to multipart/form-data
     *
     * @param requestURL
     * @throws IOException
     */
    public MultipartUtilityV2(String requestURL)
            throws IOException {

        // creates a unique boundary based on time stamp
        URL url = new URL(requestURL);
        httpConn = (HttpURLConnection) url.openConnection();
        httpConn.setUseCaches(false);
        httpConn.setDoOutput(true); // indicates POST method
        httpConn.setDoInput(true);

        httpConn.setRequestMethod("POST");
        httpConn.setRequestProperty("Connection", "Keep-Alive");
        httpConn.setRequestProperty("Cache-Control", "no-cache");
        httpConn.setRequestProperty(
                "Content-Type", "multipart/form-data;boundary=" + this.boundary);

        request =  new DataOutputStream(httpConn.getOutputStream());
    }

    /**
     * Adds a form field to the request
     *
     * @param name  field name
     * @param value field value
     */
    public void addFormField(String name, String value)throws IOException  {
        request.writeBytes(this.twoHyphens + this.boundary + this.crlf);
        request.writeBytes("Content-Disposition: form-data; name=\"" + name + "\""+ this.crlf);
        request.writeBytes("Content-Type: text/plain; charset=UTF-8" + this.crlf);
        request.writeBytes(this.crlf);
        request.writeBytes(value+ this.crlf);
        request.flush();
    }

    /**
     * Adds a upload file section to the request
     *
     * @param fieldName  name attribute in <input type="file" name="..." />
     * @param uploadFile a File to be uploaded
     * @throws IOException
     */
    public void addFilePart(String fieldName, File uploadFile)
            throws IOException {
        String fileName = uploadFile.getName();
        request.writeBytes(this.twoHyphens + this.boundary + this.crlf);
        request.writeBytes("Content-Disposition: form-data; name=\"" +
                fieldName + "\";filename=\"" +
                fileName + "\"" + this.crlf);
        request.writeBytes(this.crlf);

        byte[] bytes = Files.readAllBytes(uploadFile.toPath());
        request.write(bytes);
    }

    /**
     * Completes the request and receives response from the server.
     *
     * @return a list of Strings as response in case the server returned
     * status OK, otherwise an exception is thrown.
     * @throws IOException
     */
    public String finish() throws IOException {
        String response ="";

        request.writeBytes(this.crlf);
        request.writeBytes(this.twoHyphens + this.boundary +
                this.twoHyphens + this.crlf);

        request.flush();
        request.close();

        // checks server's status code first
        int status = httpConn.getResponseCode();
        if (status == HttpURLConnection.HTTP_OK) {
            InputStream responseStream = new
                    BufferedInputStream(httpConn.getInputStream());

            BufferedReader responseStreamReader =
                    new BufferedReader(new InputStreamReader(responseStream));

            String line = "";
            StringBuilder stringBuilder = new StringBuilder();

            while ((line = responseStreamReader.readLine()) != null) {
                stringBuilder.append(line).append("\n");
            }
            responseStreamReader.close();

            response = stringBuilder.toString();
            httpConn.disconnect();
        } else {
            throw new IOException("Server returned non-OK status: " + status);
        }

        return response;
    }
}

Agitator

This is the solution which only works for me. Thank you very much. – Asturias

but why you don't flush at the end of addFilePart() ? i saw you did flushing for the end of addFormField() . @Agitator – Hays

How to use this in another class? pls show example – Headward

Answer 4

This answer https://mcmap.net/q/173263/-sending-files-using-post-with-httpurlconnection got me 90% of the way with uploading large files to a development Django server, but I had to use setFixedLengthStreamingMode to make it worked. That requires setting the Content-Length before writing the content, thus requiring a fairly significant rewrite of the above answer. Here's my end result

public class MultipartLargeUtility {
    private final String boundary;
    private static final String LINE_FEED = "\r\n";
    private HttpURLConnection httpConn;
    private String charset;
    private OutputStream outputStream;
    private PrintWriter writer;
    private final int maxBufferSize = 4096;
    private long contentLength = 0;
    private URL url;

    private List<FormField> fields;
    private List<FilePart> files;

    private class FormField {
        public String name;
        public String value;

        public FormField(String name, String value) {
            this.name = name;
            this.value = value;
        }
    }

    private class FilePart {
        public String fieldName;
        public File uploadFile;

        public FilePart(String fieldName, File uploadFile) {
            this.fieldName = fieldName;
            this.uploadFile = uploadFile;
        }
    }

    /**
     * This constructor initializes a new HTTP POST request with content type
     * is set to multipart/form-data
     *
     * @param requestURL
     * @param charset
     * @throws IOException
     */
    public MultipartLargeUtility(String requestURL, String charset, boolean requireCSRF)
            throws IOException {
        this.charset = charset;

        // creates a unique boundary based on time stamp
        boundary = "===" + System.currentTimeMillis() + "===";
        url = new URL(requestURL);
        fields = new ArrayList<>();
        files = new ArrayList<>();

        if (requireCSRF) {
            getCSRF();
        }
    }

    /**
     * Adds a form field to the request
     *
     * @param name  field name
     * @param value field value
     */
    public void addFormField(String name, String value)
            throws UnsupportedEncodingException {
        String fieldContent = "--" + boundary + LINE_FEED;
        fieldContent += "Content-Disposition: form-data; name=\"" + name + "\"" + LINE_FEED;
        fieldContent += "Content-Type: text/plain; charset=" + charset + LINE_FEED;
        fieldContent += LINE_FEED;
        fieldContent += value + LINE_FEED;
        contentLength += fieldContent.getBytes(charset).length;
        fields.add(new FormField(name, value));
    }

    /**
     * Adds a upload file section to the request
     *
     * @param fieldName  name attribute in <input type="file" name="..." />
     * @param uploadFile a File to be uploaded
     * @throws IOException
     */
    public void addFilePart(String fieldName, File uploadFile)
            throws IOException {
        String fileName = uploadFile.getName();

        String fieldContent = "--" + boundary + LINE_FEED;
        fieldContent += "Content-Disposition: form-data; name=\"" + fieldName
                + "\"; filename=\"" + fileName + "\"" + LINE_FEED;
        fieldContent += "Content-Type: "
                + URLConnection.guessContentTypeFromName(fileName) + LINE_FEED;
        fieldContent += "Content-Transfer-Encoding: binary" + LINE_FEED;
        fieldContent += LINE_FEED;
        // file content would go here
        fieldContent += LINE_FEED;
        contentLength += fieldContent.getBytes(charset).length;
        contentLength += uploadFile.length();
        files.add(new FilePart(fieldName, uploadFile));
    }

    /**
     * Adds a header field to the request.
     *
     * @param name  - name of the header field
     * @param value - value of the header field
     */
    //public void addHeaderField(String name, String value) {
    //    writer.append(name + ": " + value).append(LINE_FEED);
    //    writer.flush();
    //}

    /**
     * Completes the request and receives response from the server.
     *
     * @return a list of Strings as response in case the server returned
     * status OK, otherwise an exception is thrown.
     * @throws IOException
     */
    public List<String> finish() throws IOException {
        List<String> response = new ArrayList<String>();
        String content = "--" + boundary + "--" + LINE_FEED;
        contentLength += content.getBytes(charset).length;

        if (!openConnection()) {
            return response;
        }

        writeContent();

        // checks server's status code first
        int status = httpConn.getResponseCode();
        if (status == HttpURLConnection.HTTP_OK) {
            BufferedReader reader = new BufferedReader(new InputStreamReader(
                    httpConn.getInputStream()));
            String line = null;
            while ((line = reader.readLine()) != null) {
                response.add(line);
            }
            reader.close();
            httpConn.disconnect();
        } else {
            throw new IOException("Server returned non-OK status: " + status);
        }
        return response;
    }

    private boolean getCSRF()
            throws IOException {
        /// First, need to get CSRF token from server
        /// Use GET request to get the token
        CookieManager cookieManager = new CookieManager();
        CookieHandler.setDefault(cookieManager);
        HttpURLConnection conn = null;

        conn = (HttpURLConnection) url.openConnection();

        conn.setUseCaches(false); // Don't use a Cached Copy
        conn.setRequestMethod("GET");
        conn.setRequestProperty("Connection", "Keep-Alive");
        conn.getContent();
        conn.disconnect();

        /// parse the returned object for the CSRF token
        CookieStore cookieJar = cookieManager.getCookieStore();
        List<HttpCookie> cookies = cookieJar.getCookies();
        String csrf = null;
        for (HttpCookie cookie : cookies) {
            Log.d("cookie", "" + cookie);
            if (cookie.getName().equals("csrftoken")) {
                csrf = cookie.getValue();
                break;
            }
        }
        if (csrf == null) {
            Log.d(TAG, "Unable to get CSRF");
            return false;
        }
        Log.d(TAG, "Received cookie: " + csrf);

        addFormField("csrfmiddlewaretoken", csrf);
        return true;
    }

    private boolean openConnection()
            throws IOException {
        httpConn = (HttpURLConnection) url.openConnection();
        httpConn.setUseCaches(false);
        httpConn.setDoOutput(true);    // indicates POST method
        httpConn.setDoInput(true);
        //httpConn.setRequestProperty("Accept-Encoding", "identity");
        httpConn.setFixedLengthStreamingMode(contentLength);
        httpConn.setRequestProperty("Connection", "Keep-Alive");
        httpConn.setRequestProperty("Content-Type",
                "multipart/form-data; boundary=" + boundary);
        outputStream = new BufferedOutputStream(httpConn.getOutputStream());
        writer = new PrintWriter(new OutputStreamWriter(outputStream, charset),
                true);
        return true;
    }

    private void writeContent()
            throws IOException {

        for (FormField field : fields) {
            writer.append("--" + boundary).append(LINE_FEED);
            writer.append("Content-Disposition: form-data; name=\"" + field.name + "\"")
                    .append(LINE_FEED);
            writer.append("Content-Type: text/plain; charset=" + charset).append(
                    LINE_FEED);
            writer.append(LINE_FEED);
            writer.append(field.value).append(LINE_FEED);
            writer.flush();
        }

        for (FilePart filePart : files) {
            String fileName = filePart.uploadFile.getName();
            writer.append("--" + boundary).append(LINE_FEED);
            writer.append(
                    "Content-Disposition: form-data; name=\"" + filePart.fieldName
                            + "\"; filename=\"" + fileName + "\"")
                    .append(LINE_FEED);
            writer.append(
                    "Content-Type: "
                            + URLConnection.guessContentTypeFromName(fileName))
                    .append(LINE_FEED);
            writer.append("Content-Transfer-Encoding: binary").append(LINE_FEED);
            writer.append(LINE_FEED);
            writer.flush();

            FileInputStream inputStream = new FileInputStream(filePart.uploadFile);
            int bufferSize = Math.min(inputStream.available(), maxBufferSize);
            byte[] buffer = new byte[bufferSize];
            int bytesRead = -1;
            while ((bytesRead = inputStream.read(buffer, 0, bufferSize)) != -1) {
                outputStream.write(buffer, 0, bytesRead);
            }
            outputStream.flush();
            inputStream.close();
            writer.append(LINE_FEED);
            writer.flush();
        }

        writer.append("--" + boundary + "--").append(LINE_FEED);
        writer.close();
    }
}

Usage is largely the same as in the above answer, but I've included CSRF support that Django uses by default with forms

boolean useCSRF = true;
MultipartLargeUtility multipart = new MultipartLargeUtility(url, "UTF-8",useCSRF);
multipart.addFormField("param1","value");
multipart.addFilePart("filefield",new File("/path/to/file"));
List<String> response = multipart.finish();
Log.w(TAG,"SERVER REPLIED:");
for(String line : response) {
    Log.w(TAG, "Upload Files Response:::" + line);
}

Radio

Hi Nick, how do i read the posted file at the receiving end? i want to read the file and write to disk. – Skean

Answer 5

based on Mihai's solution, if anyone has the problem of saving images on the server like what happened on my server. change the Bitmap to bytebuffer part to :

ByteArrayOutputStream bos = new ByteArrayOutputStream();
        bitmap.compress(Bitmap.CompressFormat.JPEG,100,bos);
        byte[] pixels = bos.toByteArray();

Mycetozoan

Answer 6

I haven't tested this, but you might try using PipedInputStream and PipedOutputStream. It might look something like:

final Bitmap bmp = … // your bitmap

// Set up Piped streams
final PipedOutputStream pos = new PipedOutputStream(new ByteArrayOutputStream());
final PipedInputStream pis = new PipedInputStream(pos);

// Send bitmap data to the PipedOutputStream in a separate thread
new Thread() {
    public void run() {
        bmp.compress(Bitmap.CompressFormat.PNG, 100, pos);
    }
}.start();

// Send POST request
try {
    // Construct InputStreamEntity that feeds off of the PipedInputStream
    InputStreamEntity reqEntity = new InputStreamEntity(pis, -1);

    HttpClient httpclient = new DefaultHttpClient();
    HttpPost httppost = new HttpPost(url);
    reqEntity.setContentType("binary/octet-stream");
    reqEntity.setChunked(true);
    httppost.setEntity(reqEntity);
    HttpResponse response = httpclient.execute(httppost);
} catch (Exception e) {
    e.printStackTrace()
}

Talkingto

Answer 7

Here is what i did for uploading photo using post request.

public void uploadFile(int directoryID, String filePath) {
    Bitmap bitmapOrg = BitmapFactory.decodeFile(filePath);
    ByteArrayOutputStream bao = new ByteArrayOutputStream();

    String upload_url = BASE_URL + UPLOAD_FILE;
    bitmapOrg.compress(Bitmap.CompressFormat.JPEG, 90, bao);

    byte[] data = bao.toByteArray();

    HttpClient httpClient = new DefaultHttpClient();
    HttpPost postRequest = new HttpPost(upload_url);
    MultipartEntity entity = new MultipartEntity(HttpMultipartMode.BROWSER_COMPATIBLE);

    try {
        // Set Data and Content-type header for the image
        FileBody fb = new FileBody(new File(filePath), "image/jpeg");
        StringBody contentString = new StringBody(directoryID + "");

        entity.addPart("file", fb);
        entity.addPart("directory_id", contentString);
        postRequest.setEntity(entity);

        HttpResponse response = httpClient.execute(postRequest);
        // Read the response
        String jsonString = EntityUtils.toString(response.getEntity());
        Log.e("response after uploading file ", jsonString);

    } catch (Exception e) {
        Log.e("Error in uploadFile", e.getMessage());
    }
}

NOTE: This code requires libraries so Follow the instructions here in order to get the libraries.

Stonewall

It's good to know that there is a way to make use of the latest version of HttpClient (your link is outdated. Use this one instead), which the Android guys are keeping only for backwards compatibility, rather than use the built in HttpURLConnection. On the other hand, it looks like it requires more work to set it up, so maybe it's not worth the effort. – Injunction

Answer 8

I tried the solutions above and none worked for me out of the box.

However http://www.baeldung.com/httpclient-post-http-request. Line 6 POST Multipart Request worked within seconds

public void whenSendMultipartRequestUsingHttpClient_thenCorrect() 
  throws ClientProtocolException, IOException {
    CloseableHttpClient client = HttpClients.createDefault();
    HttpPost httpPost = new HttpPost("http://www.example.com");

    MultipartEntityBuilder builder = MultipartEntityBuilder.create();
    builder.addTextBody("username", "John");
    builder.addTextBody("password", "pass");
    builder.addBinaryBody("file", new File("test.txt"),
      ContentType.APPLICATION_OCTET_STREAM, "file.ext");

    HttpEntity multipart = builder.build();
    httpPost.setEntity(multipart);

    CloseableHttpResponse response = client.execute(httpPost);
    client.close();
}

Antechoir