Java 9 HttpClient send a multipart/form-data request

Question

Below is a form:

<form action="/example/html5/demo_form.asp" method="post" 
enctype=”multipart/form-data”>
   <input type="file" name="img" />
   <input type="text" name=username" value="foo"/>
   <input type="submit" />
</form>

when will submit this form, the request will look like this:

POST /example/html5/demo_form.asp HTTP/1.1
Host: 10.143.47.59:9093
Connection: keep-alive
Content-Length: 326
Accept: application/json, text/javascript, */*; q=0.01
Origin: http://10.143.47.59:9093
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryEDKBhMZFowP9Leno
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8,zh-CN;q=0.6,zh;q=0.4

Request Payload
------WebKitFormBoundaryEDKBhMZFowP9Leno
Content-Disposition: form-data; name="username"

foo
------WebKitFormBoundaryEDKBhMZFowP9Leno
Content-Disposition: form-data; name="img"; filename="out.txt"
Content-Type: text/plain


------WebKitFormBoundaryEDKBhMZFowP9Leno--

please pay attention to the "Request Payload", you can see the two params in the form, the username and the img(form-data; name="img"; filename="out.txt"), and the finename is the real file name(or path) in your filesystem, you will receive the file by name(not filename) in your backend(such as spring controller).
if we use Apache Httpclient to simulate the request, we will write such code:

MultipartEntity mutiEntity = newMultipartEntity();
File file = new File("/path/to/your/file");
mutiEntity.addPart("username",new StringBody("foo", Charset.forName("utf-8")));
mutiEntity.addPart("img", newFileBody(file)); //img is name, file is path

But in java 9, We could write such code:

HttpClient client = HttpClient.newHttpClient();
HttpRequest request = HttpRequest.
        newBuilder(new URI("http:///example/html5/demo_form.asp"))
       .method("post",HttpRequest.BodyProcessor.fromString("foo"))
       .method("post", HttpRequest.BodyProcessor.fromFile(Paths.get("/path/to/your/file")))
       .build();
HttpResponse response = client.send(request, HttpResponse.BodyHandler.asString());
System.out.println(response.body());

Now you see, how could I set the "name" of the param?

Gusman

Could you share a sample API call made as you click on the button. You can monitor the same using Network settings in the Inspection section of the browser. – Vannoy

Hi, I know how to monitor the Network request and I know how to send such request using HttpClient Httpclient. What confused me is how to do it with the Httpclient in Java 9. – Gusman

I mean I know how to send such request using "Apache" Httpclient. – Gusman

Have updated the answer. The util used there is just for the part of converting the file input as a byte array which can be a custom implementation as well. – Vannoy

Thank you very much for your help. – Gusman

look no further, here is what we are really looking for – Aspiration

Answer 1

I wanted to do this for a project without having to pull in the Apache client, so I wrote a MultiPartBodyPublisher (Java 11, fyi):

import java.io.IOException;
import java.io.InputStream;
import java.io.UncheckedIOException;
import java.net.http.HttpRequest;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.*;
import java.util.function.Supplier;

public class MultiPartBodyPublisher {
    private List<PartsSpecification> partsSpecificationList = new ArrayList<>();
    private String boundary = UUID.randomUUID().toString();

    public HttpRequest.BodyPublisher build() {
        if (partsSpecificationList.size() == 0) {
            throw new IllegalStateException("Must have at least one part to build multipart message.");
        }
        addFinalBoundaryPart();
        return HttpRequest.BodyPublishers.ofByteArrays(PartsIterator::new);
    }

    public String getBoundary() {
        return boundary;
    }

    public MultiPartBodyPublisher addPart(String name, String value) {
        PartsSpecification newPart = new PartsSpecification();
        newPart.type = PartsSpecification.TYPE.STRING;
        newPart.name = name;
        newPart.value = value;
        partsSpecificationList.add(newPart);
        return this;
    }

    public MultiPartBodyPublisher addPart(String name, Path value) {
        PartsSpecification newPart = new PartsSpecification();
        newPart.type = PartsSpecification.TYPE.FILE;
        newPart.name = name;
        newPart.path = value;
        partsSpecificationList.add(newPart);
        return this;
    }

    public MultiPartBodyPublisher addPart(String name, Supplier<InputStream> value, String filename, String contentType) {
        PartsSpecification newPart = new PartsSpecification();
        newPart.type = PartsSpecification.TYPE.STREAM;
        newPart.name = name;
        newPart.stream = value;
        newPart.filename = filename;
        newPart.contentType = contentType;
        partsSpecificationList.add(newPart);
        return this;
    }

    private void addFinalBoundaryPart() {
        PartsSpecification newPart = new PartsSpecification();
        newPart.type = PartsSpecification.TYPE.FINAL_BOUNDARY;
        newPart.value = "--" + boundary + "--";
        partsSpecificationList.add(newPart);
    }

    static class PartsSpecification {

        public enum TYPE {
            STRING, FILE, STREAM, FINAL_BOUNDARY
        }

        PartsSpecification.TYPE type;
        String name;
        String value;
        Path path;
        Supplier<InputStream> stream;
        String filename;
        String contentType;

    }

    class PartsIterator implements Iterator<byte[]> {

        private Iterator<PartsSpecification> iter;
        private InputStream currentFileInput;

        private boolean done;
        private byte[] next;

        PartsIterator() {
            iter = partsSpecificationList.iterator();
        }

        @Override
        public boolean hasNext() {
            if (done) return false;
            if (next != null) return true;
            try {
                next = computeNext();
            } catch (IOException e) {
                throw new UncheckedIOException(e);
            }
            if (next == null) {
                done = true;
                return false;
            }
            return true;
        }

        @Override
        public byte[] next() {
            if (!hasNext()) throw new NoSuchElementException();
            byte[] res = next;
            next = null;
            return res;
        }

        private byte[] computeNext() throws IOException {
            if (currentFileInput == null) {
                if (!iter.hasNext()) return null;
                PartsSpecification nextPart = iter.next();
                if (PartsSpecification.TYPE.STRING.equals(nextPart.type)) {
                    String part =
                            "--" + boundary + "\r\n" +
                            "Content-Disposition: form-data; name=" + nextPart.name + "\r\n" +
                            "Content-Type: text/plain; charset=UTF-8\r\n\r\n" +
                            nextPart.value + "\r\n";
                    return part.getBytes(StandardCharsets.UTF_8);
                }
                if (PartsSpecification.TYPE.FINAL_BOUNDARY.equals(nextPart.type)) {
                    return nextPart.value.getBytes(StandardCharsets.UTF_8);
                }
                String filename;
                String contentType;
                if (PartsSpecification.TYPE.FILE.equals(nextPart.type)) {
                    Path path = nextPart.path;
                    filename = path.getFileName().toString();
                    contentType = Files.probeContentType(path);
                    if (contentType == null) contentType = "application/octet-stream";
                    currentFileInput = Files.newInputStream(path);
                } else {
                    filename = nextPart.filename;
                    contentType = nextPart.contentType;
                    if (contentType == null) contentType = "application/octet-stream";
                    currentFileInput = nextPart.stream.get();
                }
                String partHeader =
                        "--" + boundary + "\r\n" +
                        "Content-Disposition: form-data; name=" + nextPart.name + "; filename=" + filename + "\r\n" +
                        "Content-Type: " + contentType + "\r\n\r\n";
                return partHeader.getBytes(StandardCharsets.UTF_8);
            } else {
                byte[] buf = new byte[8192];
                int r = currentFileInput.read(buf);
                if (r > 0) {
                    byte[] actualBytes = new byte[r];
                    System.arraycopy(buf, 0, actualBytes, 0, r);
                    return actualBytes;
                } else {
                    currentFileInput.close();
                    currentFileInput = null;
                    return "\r\n".getBytes(StandardCharsets.UTF_8);
                }
            }
        }
    }
}

You can use it approximately like so:

MultiPartBodyPublisher publisher = new MultiPartBodyPublisher()
       .addPart("someString", "foo")
       .addPart("someInputStream", () -> this.getClass().getResourceAsStream("test.txt"), "test.txt", "text/plain")
       .addPart("someFile", pathObject);
HttpRequest request = HttpRequest.newBuilder()
       .uri(URI.create("https://www.example.com/dosomething"))
       .header("Content-Type", "multipart/form-data; boundary=" + publisher.getBoundary())
       .timeout(Duration.ofMinutes(1))
       .POST(publisher.build())
       .build();

Note that addPart for input streams actually takes a Supplier<InputStream> and not just an InputStream.

Infiltration

Consider putting this on github. – Malek

@Infiltration you deserve a beer! this is absolutely beautiful – Aspiration

@Infiltration now that I've looked at it more closely and used it - it can be greatly simplified, I hope to have time to post it on github. You also seems to me to have a problem when there would 2 files to be uploaded, since I have changed the code a lot, can't ell for sure. – Aspiration

I do use it to upload multiple files in a single request, though, so that should work. Curious to see your changes...I haven't thought about this much recently. – Infiltration

This is great indeed! I had to modify the last bit to get the streaming of file bytes to work. byte[] bytes = currentFileInput.readAllBytes(); currentFileInput.close(); currentFileInput = null; byte[] actual = new byte[bytes.length + 2]; byte[] newline = "\r\n".getBytes(StandardCharsets.UTF_8); System.arraycopy(bytes, 0, actual, 0, bytes.length); System.arraycopy(newline, 0, actual, bytes.length, newline.length); return actual; Sorry for the poor formatting. – Lightning

This does not work. Only the first part is sent, as per the java.net.http.HttpRequest.BodyPublishers#ofByteArrays doc says: . Each attempt to send the request results in one invocation of the Iterable while this implementation expect the iterable to be exhausted in the first attempt. – Fermium

Answer 2

You can use Methanol. It contains a MultipartBodyPublisher with a convenient and easy to use MultipartBodyPublisher.Builder. Here is an example using it (JDK11 or later is required):

var multipartBody = MultipartBodyPublisher.newBuilder()
    .textPart("foo", "foo_text")
    .filePart("bar", Path.of("path/to/file.txt"))
    .formPart("baz", BodyPublishers.ofInputStream(() -> ...))
    .build();
var request = HttpRequest.newBuilder()
    .uri(URI.create("https://example.com/"))
    .POST(multipartBody)
    .build();

Note that you can add any BodyPublisher or HttpHeaders you want. Check out the docs for more info.

Kessinger

Answer 3

A direction in which you can attain making a multiform-data call could be as follows:

BodyProcessor can be used with their default implementations or else a custom implementation can also be used. Few of the ways to use them are :

1. Read the processor via a string as :

   HttpRequest.BodyProcessor dataProcessor = HttpRequest.BodyProcessor.fromString("{\"username\":\"foo\"}")
   

2. Creating a processor from a file using its path

   Path path = Paths.get("/path/to/your/file"); // in your case path to 'img'
   HttpRequest.BodyProcessor fileProcessor = HttpRequest.BodyProcessor.fromFile(path);
   

OR

3. You can convert the file input to a byte array using the apache.commons.lang(or a custom method you can come up with) to add a small util like :

   org.apache.commons.fileupload.FileItem file;
   
   org.apache.http.HttpEntity multipartEntity = org.apache.http.entity.mime.MultipartEntityBuilder.create()
          .addPart("username",new StringBody("foo", Charset.forName("utf-8")))
          .addPart("img", newFileBody(file))
          .build();
   multipartEntity.writeTo(byteArrayOutputStream);
   byte[] bytes = byteArrayOutputStream.toByteArray();
   

   and then the byte[] can be used with BodyProcessor as:

   HttpRequest.BodyProcessor byteProcessor = HttpRequest.BodyProcessor.fromByteArray();
   

---

Further, you can create the request as :

HttpRequest request = HttpRequest.newBuilder()
            .uri(new URI("http:///example/html5/demo_form.asp"))
            .headers("Content-Type","multipart/form-data","boundary","boundaryValue") // appropriate boundary values
            .POST(dataProcessor)
            .POST(fileProcessor)
            .POST(byteProcessor) //self-sufficient
            .build();

---

The response for the same can be handled as a file and with a new HttpClient using

HttpResponse.BodyHandler bodyHandler = HttpResponse.BodyHandler.asFile(Paths.get("/path"));

HttpClient client = HttpClient.newBuilder().build();

as:

HttpResponse response = client.send(request, bodyHandler);
System.out.println(response.body());

Vannoy

I really apprecate your answer. But could you tell the name of the file? Just like "Content-Disposition: form-data; name="myfile"; filename="/path/to/your/file" " – Gusman

yes, in the form it is 'img', but where do you set it in java code ? – Gusman

"Content-Disposition: form-data; name="myfile"; filename="/path/to/your/file" ---I mean the name, not filename – Gusman

Unfortunately calling POST multiple times does not work. You can only set a single processor. (at least in the latest version of the client) – Kessiah

@Kapep Could possibly be, I haven't given it a try myself either. These were just a direction to make use of. The byteProcessor though, should be self-sufficient I thought. – Vannoy

Answer 4

It is possible to use multipart/form-data or any other content type - but you have to encode the body in the correct format yourself. The client itself does not do any encoding based on the content type.

That means your best option is to use another HTTP client the like Apache HttpComponents client or only use the encoder of another library like in the example of @nullpointer's answer.

---

If you do encode the body yourself, note that you can't call methods like POST more than once. POST simply sets the BodyProcessor and calling it again will just override any previously set processors. You have to implement one processor that produces the whole body in the correct format.

For multipart/form-data that means:

1. Set the boundary header to an appropriate value

2. Encode each parameter so that it looks like in your example. Basically something like this for text input:

   boundary + "\nContent-Disposition: form-data; name=\"" + name + "\"\n\n" + value + "\n"
   

   Here, the name refers to the name attribute in the HTML form. For the file input in the question, this would img and the value would be the encoded file content.

Kessiah

Answer 5

I struggled with this problem for a while, even after seeing and reading this page. But, using the answers on this page to point me in the right direction, reading more about multipart forms and boundaries, and tinkering around, I was able to create a working solution.

The gist of the solution is to use Apache's MultipartEntityBuilder to create the entity and its boundaries (HttpExceptionBuilder is a homegrown class):

import java.io.BufferedInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.util.Optional;
import java.util.function.Supplier;

import org.apache.commons.lang3.Validate;
import org.apache.http.HttpEntity;
import org.apache.http.entity.BufferedHttpEntity;
import org.apache.http.entity.ContentType;
import org.apache.http.entity.mime.MultipartEntityBuilder;

/**
 * Class containing static helper methods pertaining to HTTP interactions.
 */
public class HttpUtils {
    public static final String MULTIPART_FORM_DATA_BOUNDARY = "ThisIsMyBoundaryThereAreManyLikeItButThisOneIsMine";

    /**
     * Creates an {@link HttpEntity} from a {@link File}, loading it into a {@link BufferedHttpEntity}.
     *
     * @param file     the {@link File} from which to create an {@link HttpEntity}
     * @param partName an {@link Optional} denoting the name of the form data; defaults to {@code data}
     * @return an {@link HttpEntity} containing the contents of the provided {@code file}
     * @throws NullPointerException  if {@code file} or {@code partName} is null
     * @throws IllegalStateException if {@code file} does not exist
     * @throws HttpException         if file cannot be found or {@link FileInputStream} cannot be created
     */
    public static HttpEntity getFileAsBufferedMultipartEntity(final File file, final Optional<String> partName) {
        Validate.notNull(file, "file cannot be null");
        Validate.validState(file.exists(), "file must exist");
        Validate.notNull(partName, "partName cannot be null");

        final HttpEntity entity;
        final BufferedHttpEntity bufferedHttpEntity;

        try (final FileInputStream fis = new FileInputStream(file);
                final BufferedInputStream bis = new BufferedInputStream(fis)) {
            entity = MultipartEntityBuilder.create().setBoundary(MULTIPART_FORM_DATA_BOUNDARY)
                    .addBinaryBody(partName.orElse("data"), bis, ContentType.APPLICATION_OCTET_STREAM, file.getName())
                    .setContentType(ContentType.MULTIPART_FORM_DATA).build();

            try {
                bufferedHttpEntity = new BufferedHttpEntity(entity);
            } catch (final IOException e) {
                throw HttpExceptionBuilder.create().withMessage("Unable to create BufferedHttpEntity").withThrowable(e)
                        .build();
            }
        } catch (final FileNotFoundException e) {
            throw HttpExceptionBuilder.create()
                    .withMessage("File does not exist or is not readable: %s", file.getAbsolutePath()).withThrowable(e)
                    .build();
        } catch (final IOException e) {
            throw HttpExceptionBuilder.create()
                    .withMessage("Unable to create multipart entity from file: %s", file.getAbsolutePath())
                    .withThrowable(e).build();
        }

        return bufferedHttpEntity;
    }

    /**
     * Returns a {@link Supplier} of {@link InputStream} containing the content of the provided {@link HttpEntity}. This
     * method closes the {@code InputStream}.
     *
     * @param entity the {@link HttpEntity} from which to get an {@link InputStream}
     * @return an {@link InputStream} containing the {@link HttpEntity#getContent() content}
     * @throws NullPointerException if {@code entity} is null
     * @throws HttpException        if something goes wrong
     */
    public static Supplier<? extends InputStream> getInputStreamFromHttpEntity(final HttpEntity entity) {
        Validate.notNull(entity, "entity cannot be null");

        return () -> {
            try (final InputStream is = entity.getContent()) {
                return is;
            } catch (final UnsupportedOperationException | IOException e) {
                throw HttpExceptionBuilder.create().withMessage("Unable to get InputStream from HttpEntity")
                        .withThrowable(e).build();
            }
        };
    }
}

And then a method that uses these helper methods:

private String doUpload(final File uploadFile, final String filePostUrl) {
    assert uploadFile != null : "uploadFile cannot be null";
    assert uploadFile.exists() : "uploadFile must exist";
    assert StringUtils.notBlank(filePostUrl, "filePostUrl cannot be blank");

    final URI uri = URI.create(filePostUrl);
    final HttpEntity entity = HttpUtils.getFileAsBufferedMultipartEntity(uploadFile, Optional.of("partName"));
    final String response;

    try {
        final Builder requestBuilder = HttpRequest.newBuilder(uri)
                .POST(BodyPublisher.fromInputStream(HttpUtils.getInputStreamFromHttpEntity(entity)))
                .header("Content-Type", "multipart/form-data; boundary=" + HttpUtils.MULTIPART_FORM_DATA_BOUNDARY);

        response = this.httpClient.send(requestBuilder.build(), BodyHandler.asString());
    } catch (InterruptedException | ExecutionException e) {
        throw HttpExceptionBuilder.create().withMessage("Unable to get InputStream from HttpEntity")
                    .withThrowable(e).build();
    }

    LOGGER.info("Http Response: {}", response);
    return response;
}

Praxiteles

ThisIsMyBoundaryThereAreManyLikeItButThisOneIsMine made my day – Gunstock

Answer 6

While the correct answer is full-blown implementation and might be correct, it did not work for me.

My solution took inspiration from here. I just cleaned up for my use case not required parts. Me, personally, use multipart form for only uploading picture or zip file (singular). The code:

    public static HttpRequest buildMultiformRequest(byte[] body) {
        String boundary = "-------------" + UUID.randomUUID().toString();
        Map<String, byte[]> data = Map.of("formFile", body);

        return HttpRequest.newBuilder()
                .uri(URI.create(<URL>))
                .POST(HttpRequest.BodyPublishers.ofByteArrays(buildMultipartData(data, boundary, "filename.jpeg", MediaType.IMAGE_JPEG_VALUE)))
                .header("Content-Type", "multipart/form-data; boundary=" + boundary)
                .header("Accept", MediaType.APPLICATION_JSON_VALUE)
                .timeout(Duration.of(5, ChronoUnit.SECONDS))
                .build();
    }

    public static ArrayList<byte[]> buildMultipartData(Map<String, byte[]> data, String boundary, String filename, String mediaType) {
        var byteArrays = new ArrayList<byte[]>();
        var separator = ("--" + boundary + "\r\nContent-Disposition: form-data; name=").getBytes(StandardCharsets.UTF_8);

        for (var entry : data.entrySet()) {
            byteArrays.add(separator);
            byteArrays.add(("\"" + entry.getKey() + "\"; filename=\"" + filename + "\"\r\nContent-Type:" + mediaType + "\r\n\r\n").getBytes(StandardCharsets.UTF_8));
            byteArrays.add(entry.getValue());
            byteArrays.add("\r\n".getBytes(StandardCharsets.UTF_8));
        }

        byteArrays.add(("--" + boundary + "--").getBytes(StandardCharsets.UTF_8));
        return byteArrays;
    }

Curd

Answer 7

The following worked for me, namely to create a raw HTTP body as a String in memory and then use the standard BodyPublisher.ofString:

The following link shows what the body should look like: https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/POST

String data = "--boundary\nContent-Disposition: form-data; name=\"type\"\r\n\r\nserverless";
byte[] fileContents = Files.readAllBytes(f.toPath());
data += "\r\n--boundary\nContent-Disposition: form-data; name=\"filename\"; filename=\""
        + f.getName() + "\"\r\n\r\n" + new String(fileContents, StandardCharsets.ISO_8859_1); // iso-8859-1 is http default
data += "\r\n--boundary--"; // end boundary

HttpRequest.BodyPublisher bodyPublisher = HttpRequest.BodyPublishers.ofString(data, StandardCharsets.ISO_8859_1);

HttpRequest request = HttpRequest.newBuilder()
                    .uri(uri)
                    .setHeader("Content-Type", "multipart/form-data;boundary=\"boundary\"")
                    .POST(bodyPublisher).build();
HttpResponse<String> response = getClient().send(request, HttpResponse.BodyHandlers.ofString());

Watch out for the \r\n instead of just say \n - I tested this with Apache Commons File Upload which expects both, probably because that's what the RFC expects.

Also note the use of ISO-8859-1 instead of UTF-8. I used this because it's the standard - I didn't test it with UTF-8 - it might work if the server is also configured that way.

getClient roughly does this:

HttpClient.newBuilder()
    .version(HttpClient.Version.HTTP_1_1)
    .connectTimeout(Duration.ofSeconds(20))
    .build()

Wintery

Answer 8

Simply you can create a multipart entity (request body form-data) by using MultipartEntityBuilder:

// Create a multipart entity (form-data)
HttpEntity entity = MultipartEntityBuilder.create()
        .addTextBody("email", "[email protected]")
        .addTextBody("firstName", "firstName")
        .addTextBody("lastName", "lastName")
        .build();

HttpUriRequest request = RequestBuilder.post(myUrl)
        .setEntity(entity)
        .build();

MyResponse myResponse = httpClient.execute(request, new MyRequestDataFeedResponseHandler());



// define a class which will handle the HTTP response
class MyRequestDataFeedResponseHandler implements ResponseHandler<MyResponse> {

    public MyResponse handleResponse(HttpResponse httpResponse) throws IOException {
        // ...
    }

}

Arlinearlington