Posting a File and Associated Data to a RESTful WebService preferably as JSON

Question

In an application I am developing RESTful API and we want the client to send data as JSON. Part of this application requires the client to upload a file (usually an image) as well as information about the image.

I'm having a hard time tracking down how this happens in a single request. Is it possible to Base64 the file data into a JSON string? Am I going to need to perform 2 posts to the server? Should I not be using JSON for this?

As a side note, we're using Grails on the backend and these services are accessed by native mobile clients (iPhone, Android, etc), if any of that makes a difference.

Cabaret

Send the metadata in the URL query string, instead of JSON. – Overwrought

Another related SO question. – Poplar

Does this answer your question? How do I upload a file with metadata using a REST web service? – Lordling

Answer 1

I asked a similar question here:

How do I upload a file with metadata using a REST web service?

You basically have three choices:

1. Base64 encode the file, at the expense of increasing the data size by around 33%, and add processing overhead in both the server and the client for encoding/decoding.
2. Send the file first in a multipart/form-data POST, and return an ID to the client. The client then sends the metadata with the ID, and the server re-associates the file and the metadata.
3. Send the metadata first, and return an ID to the client. The client then sends the file with the ID, and the server re-associates the file and the metadata.

Staley

If I chose option 1, do I just include the Base64 content inside the JSON string? {file:'234JKFDS#$@#$MFDDMS....', name:'somename'...} Or is there something more to it? – Cabaret

Marked this as the answer because the linked question basically covers everything I need. Thanks Daniel. – Cabaret

Gregg, exactly as you've said, you would just include it as a property, and the value would be the base64-encoded string. This is probably the easiest method to go with, but might not be practical depending on the file size. For example, for our application, we need to send iPhone images that are 2-3 MB each. An increase of 33% is not acceptable. If you're sending only small 20KB images, that overhead might be more acceptable. – Staley

I should also mention that the base64 encoding/decoding will also take some processing time. It might be the easiest thing to do, but it's certainly not the best. – Staley

If one gives up the JSON part, how would be sending all as multipart/form-data as a solution? I'm asking because I can't see why I should use JSON at this point (unless I encode the file in JSON+Base64). – Demagogue

I have the exact problem but I'm at the design stage of the API so I'm not tied to the JSON part. @Demagogue I'm wondering whether you've had success with using multipart/form-data for sending both data (string) and file in the same request and process it that way? – Muliebrity

@Muliebrity sorry, I didn't even try, as far as I remember. I used json. – Demagogue

json with base64? hmm.. I'm thinking about sticking to multipart/form – Muliebrity

Option 4: Modify your WebService API to accept either JSON or multipart/form-data input. – Rub

I've been on this for the last few hours and finally came across multipart/form-data. How do you send the file? It's just a REST API service so how do you upload a file when there is no upload field. – Amor

I'd like to add a 4th option here - to remove the need for the sending order of options #2 and #3: Create a UUID on the client. Send the file with this UUID / Send the metadata with this UUID (in whatever order). The 'feature' of this approach is the client is responsible for creating the UUID - but this is often needed anyway to help with synchronisation. – Wayne

Why it is deny to use multipart/form-data in one request? – Dolor

For me fourth option is, in multipart/form-data add another form data of type 'TEXT' and use the stringified meta data. – Ominous

Most of my cases point to #3 as best option, because I first send the metadata and then get an ID connected to another file (my image) to send to. Hence, the server expects me to send something to an ID he provides me with. This way I am able to place some server-side logic behind this. – Wylie

@Wylie I would say option 2 is more safe. Suppose you are managing a form with some attachments. You upload the file and receive an ID from the server. If file upload succeeds, you update the form with the file id and send a save request for the form. In this way, if the file fails, you never update the form. The other way around you have the risk of updating the form and then failing to upload the file. – Forenoon

Maybe combine two and three? Get an id, send the file and then signal to the server "I am done transmitting". Could be helpful with really large files. Some kind of checksum validation could also be added. – Sorenson

how about generating a UUID on client itself, then posting two requests one for the data, the other multipart/form-data just for the file – Sugar

if you have to choose between 2 and 3.: might be more effective to send the file in first place, return an ID and include this in the subsequent metadata request. In this way you can avoid to send metadata when file upload is failed, but most depends on the use case and which inconsistency (file w/o metadata or metadata w/o) you can tolerate, and ability for clients to retry – Wintertime

Interesting. Why would you send the potentially large file first? What if the metadata never arrives? How long do you store the mostly useless file on server and wait for the metadata? If you send the meta first, you flag the file (or data object) as incomplete until the payload arrives. That way you can employ garbage collector and remove the incomplete files, that do not take much space anyway. You also have the rest of the form data and can ask the user to provide the payload later. (Assuming you are uploading a complex form, not just a single file). – Kuth

Why not stringify metadata JSON and send it using form data along with files in one request? – Annorah

Choice 1 is a very bad choice for memory. Your server won't be able to stream the file – Furan

Answer 2

You can send the file and data over in one request using the multipart/form-data content type:

In many applications, it is possible for a user to be presented with a form. The user will fill out the form, including information that is typed, generated by user input, or included from files that the user has selected. When the form is filled out, the data from the form is sent from the user to the receiving application.

The definition of MultiPart/Form-Data is derived from one of those applications...

From http://www.faqs.org/rfcs/rfc2388.html:

"multipart/form-data" contains a series of parts. Each part is expected to contain a content-disposition header [RFC 2183] where the disposition type is "form-data", and where the disposition contains an (additional) parameter of "name", where the value of that parameter is the original field name in the form. For example, a part might contain a header:

Content-Disposition: form-data; name="user"

with the value corresponding to the entry of the "user" field.

You can include file information or field information within each section between boundaries. I've successfully implemented a RESTful service that required the user to submit both data and a form, and multipart/form-data worked perfectly. The service was built using Java/Spring, and the client was using C#, so unfortunately I don't have any Grails examples to give you concerning how to set up the service. You don't need to use JSON in this case since each "form-data" section provides you a place to specify the name of the parameter and its value.

The good thing about using multipart/form-data is that you're using HTTP-defined headers, so you're sticking with the REST philosophy of using existing HTTP tools to create your service.

Cohune

Thanks, but my question was focused on wanting to use JSON for the request and if that was possible. I already know that I could send it the way you suggest. – Cabaret

Yeah that's essentially my response for "Should I not be using JSON for this?" Is there a specific reason why you want the client to use JSON? – Cohune

Most likely a business requirement or keeping with consistency. Of course, the ideal thing to do is accept both (form data and JSON response) based on the Content-Type HTTP header. – Staley

I apologize for what I said if it hurt some .Net developer's feeling. Although English is not my native language, it's not a valid excuse for me to say something rude about the technology itself. Using form data is awesome and if you keep using it you'll be even more awesome, too! – Ziegfeld

But in this case, how to GET both text data and image on the client side since there is one endpoint for both of them? – Lanford

Why can't one form field be the json, a second field be the multipart file? The server would just have to parse the json and go on with life. – Malcommalcontent

so, I can create an HTML form and receive a file and a series of inputs in one request. Php receives $_POST values as well as $_FILES. But I can't recreate this with JS fetch. If anyone knows how to achieve this I'll happily upvote it as an answer. – Ori

Answer 3

I know that this thread is quite old, however, I am missing here one option. If you have metadata (in any format) that you want to send along with the data to upload, you can make a single multipart/related request.

The Multipart/Related media type is intended for compound objects consisting of several inter-related body parts.

You can check RFC 2387 specification for more in-depth details.

Basically each part of such a request can have content with different type and all parts are somehow related (e.g. an image and it metadata). The parts are identified by a boundary string, and the final boundary string is followed by two hyphens.

Example:

POST /upload HTTP/1.1
Host: www.hostname.com
Content-Type: multipart/related; boundary=xyz
Content-Length: [actual-content-length]

--xyz
Content-Type: application/json; charset=UTF-8

{
    "name": "Sample image",
    "desc": "...",
    ...
}

--xyz
Content-Type: image/jpeg

[image data]
[image data]
[image data]
...
--foo_bar_baz--

Prinz

I liked your solution the best by far. Unfortunately, there appears to be no way to create mutlipart/related requests in a browser. – Fong

do you have any experience in getting clients to (especially JS ones) to communicate with the api in this way – Funkhouser

unfortunately, there's currently no reader for this kind of data on php (7.2.1) and you would have to build your own parser – Cosmogony

It's sad that servers and clients don't have good support for this. – Jetport

the solution has two problems: one is that it needs to be supported by client/server web frameworks that are used for the implementation, the second is that, if the validation of the json part fails (eg, one of the metadata is an email address), it should return an error and lead client to re-upload the file, which is expensive – Wintertime

Answer 4

Here is my approach API (i use example) - as you can see, you I don't use any file_id (uploaded file identifier to the server) in API:

1. Create photo object on server:

    POST: /projects/{project_id}/photos   
    body: { name: "some_schema.jpg", comment: "blah"}
    response: photo_id
   

2. Upload file (note that file is in singular form because it is only one per photo):

    POST: /projects/{project_id}/photos/{photo_id}/file
    body: file to upload
    response: -
   

And then for instance:

3. Read photos list

    GET: /projects/{project_id}/photos
    response: [ photo, photo, photo, ... ] (array of objects)
   

4. Read some photo details

    GET: /projects/{project_id}/photos/{photo_id}
    response: { id: 666, name: 'some_schema.jpg', comment:'blah'} (photo object)
   

5. Read photo file

    GET: /projects/{project_id}/photos/{photo_id}/file
    response: file content
   

So the conclusion is that, first you create an object (photo) by POST, and then you send second request with the file (again POST). To not have problems with CACHE in this approach we assume that we can only delete old photos and add new - no update binary photo files (because new binary file is in fact... NEW photo). However if you need to be able to update binary files and cache them, then in point 4 return also fileId and change 5 to GET: /projects/{project_id}/photos/{photo_id}/files/{fileId}.

Brightness

This seems like the more 'RESTFUL' way to achieve this. – Radarman

POST operation for newly created resources, must return location id, in simple version details of the object – Intramundane

@ivanproskuryakov why "must"? In the example above (POST in point 2) the file id is useless. Second argument (for POST in point 2) i use singular form '/file' (not '/files') so ID is not needed because path: /projects/2/photos/3/file give FULL information to identity photo file. – Villanueva

From HTTP protocol specification. w3.org/Protocols/rfc2616/rfc2616-sec10.html 10.2.2 201 Created "The newly created resource can be referenced by the URI(s) returned in the entity of the response, with the most specific URI for the resource given by a Location header field." @KamilKiełczewski (one) and (two) could be combined into one POST operation POST: /projects/{project_id}/photos Will return you location header, which could be used for GET single photo(resource*) operation GET: to get a single photo with all details CGET: to get all collection of the photos – Intramundane

@ivanproskuryakov in the quotation you used, there no word 'must'. Your idea to "combine (1) and (2) into one POST" is only alternative approach (but I don't understand how do you put in your approach other photo informations like "comment" ?). So I don't see any mistake in my approach - POST without returning ID is acceptable because this uri /projects/2/photos/3/file determine file (and any additional information about internal server file id is useless). – Villanueva

@KamilKiełczewski "SHOULD" is the correct word, pardon. The idea is not mine, just a basic approach. So. if we create a resource we need reply with information how to access newly created resource(location header) Spec: w3.org/TR/html401/interact/forms.html#h-17.13.4.2 Ex: https://mcmap.net/q/54309/-uploading-both-data-and-files-in-one-form-using-ajax – Intramundane

@ivanproskuryakov your link from w3.org/TR/html401/interact/forms.html#h-17.13.4.2 not apply to the question case which is RESTful API (ajax) - not FORM submmition. In my case (2) POST does not have to return anything. – Villanueva

@ivanproskuryakov Multipart/form-data is not approach that I describe in my above answer. I is alternative approach - not worse, not better just alternative. The people mention about it in other answers. – Villanueva

If metadata and upload are separate operations, then the endpoints have these issues: For file upload POST operation used - POST is not idempotent. PUT(idempotent) must be used since you are changing the resource without creating a new one. REST works with objects called resources. POST: “../photos/“ PUT: “../photos/{photo_id}” GET: “../photos/“ GET: “../photos/{photo_id}” PS. Separating upload into separate endpoint may lead to unpredicted behavior. restapitutorial.com/lessons/idempotency.html restful-api-design.readthedocs.io/en/latest/resources.html – Intramundane

@ivanproskuryakov Look closely i use singular form 'file' not plural form 'files' - it is not coincidence. In restful API is being used singular form for resources that are "only one" which entitles to avoid resource_id. Photo has only one file so i use singular form ( softwareengineering.stackexchange.com/a/245205 ) – Villanueva

The negative vote was my Personal feedback on your answer. In case if you have time, take a look on "Resources and Representations" from "RESTful Web APIs". I think it will help to understand why my vote on the answer was negative. amazon.com/RESTful-Web-APIs-Services-Changing-ebook/dp/… – Intramundane

@IvanProskuryakov rfc POST: "The action performed by the POST method might not result in a resource that can be identified by a URI. In this case, either 200 (OK) or 204 (No Content) is the appropriate response status, depending on whether or not the response includes an entity that describes the result." – Villanueva

True, if no resource was created then 200 or 204. Looking backwards on what’s was written I would now - say is not set in stone. Cheers! – Intramundane

Answer 5

I know this question is old, but in the last days I had searched whole web to solution this same question. I have grails REST webservices and iPhone Client that send pictures, title and description.

I don't know if my approach is the best, but is so easy and simple.

I take a picture using the UIImagePickerController and send to server the NSData using the header tags of request to send the picture's data.

NSMutableURLRequest *request = [[NSMutableURLRequest alloc] initWithURL:[NSURL URLWithString:@"myServerAddress"]];
[request setHTTPMethod:@"POST"];
[request setHTTPBody:UIImageJPEGRepresentation(picture, 0.5)];
[request setValue:@"image/jpeg" forHTTPHeaderField:@"Content-Type"];
[request setValue:@"myPhotoTitle" forHTTPHeaderField:@"Photo-Title"];
[request setValue:@"myPhotoDescription" forHTTPHeaderField:@"Photo-Description"];

NSURLResponse *response;

NSError *error;

[NSURLConnection sendSynchronousRequest:request returningResponse:&response error:&error];

At the server side, I receive the photo using the code:

InputStream is = request.inputStream

def receivedPhotoFile = (IOUtils.toByteArray(is))

def photo = new Photo()
photo.photoFile = receivedPhotoFile //photoFile is a transient attribute
photo.title = request.getHeader("Photo-Title")
photo.description = request.getHeader("Photo-Description")
photo.imageURL = "temp"    

if (photo.save()) {    

    File saveLocation = grailsAttributes.getApplicationContext().getResource(File.separator + "images").getFile()
    saveLocation.mkdirs()

    File tempFile = File.createTempFile("photo", ".jpg", saveLocation)

    photo.imageURL = saveLocation.getName() + "/" + tempFile.getName()

    tempFile.append(photo.photoFile);

} else {

    println("Error")

}

I don't know if I have problems in future, but now is working fine in production environment.

Mcmorris

I like this option of using http headers. This works especially well when there is some symmetry between the metadata and standard http headers, but you can obviously invent your own. – Feodora

Answer 6

FormData Objects: Upload Files Using Ajax

XMLHttpRequest Level 2 adds support for the new FormData interface. FormData objects provide a way to easily construct a set of key/value pairs representing form fields and their values, which can then be easily sent using the XMLHttpRequest send() method.

function AjaxFileUpload() {
    var file = document.getElementById("files");
    //var file = fileInput;
    var fd = new FormData();
    fd.append("imageFileData", file);
    var xhr = new XMLHttpRequest();
    xhr.open("POST", '/ws/fileUpload.do');
    xhr.onreadystatechange = function () {
        if (xhr.readyState == 4) {
             alert('success');
        }
        else if (uploadResult == 'success')
             alert('error');
    };
    xhr.send(fd);
}

https://developer.mozilla.org/en-US/docs/Web/API/FormData

Metralgia

Answer 7

Since the only missing example is the ANDROID example, I'll add it. This technique uses a custom AsyncTask that should be declared inside your Activity class.

private class UploadFile extends AsyncTask<Void, Integer, String> {
    @Override
    protected void onPreExecute() {
        // set a status bar or show a dialog to the user here
        super.onPreExecute();
    }

    @Override
    protected void onProgressUpdate(Integer... progress) {
        // progress[0] is the current status (e.g. 10%)
        // here you can update the user interface with the current status
    }

    @Override
    protected String doInBackground(Void... params) {
        return uploadFile();
    }

    private String uploadFile() {

        String responseString = null;
        HttpClient httpClient = new DefaultHttpClient();
        HttpPost httpPost = new HttpPost("http://example.com/upload-file");

        try {
            AndroidMultiPartEntity ampEntity = new AndroidMultiPartEntity(
                new ProgressListener() {
                    @Override
                        public void transferred(long num) {
                            // this trigger the progressUpdate event
                            publishProgress((int) ((num / (float) totalSize) * 100));
                        }
            });

            File myFile = new File("/my/image/path/example.jpg");

            ampEntity.addPart("fileFieldName", new FileBody(myFile));

            totalSize = ampEntity.getContentLength();
            httpPost.setEntity(ampEntity);

            // Making server call
            HttpResponse httpResponse = httpClient.execute(httpPost);
            HttpEntity httpEntity = httpResponse.getEntity();

            int statusCode = httpResponse.getStatusLine().getStatusCode();
            if (statusCode == 200) {
                responseString = EntityUtils.toString(httpEntity);
            } else {
                responseString = "Error, http status: "
                        + statusCode;
            }

        } catch (Exception e) {
            responseString = e.getMessage();
        }
        return responseString;
    }

    @Override
    protected void onPostExecute(String result) {
        // if you want update the user interface with upload result
        super.onPostExecute(result);
    }

}

So, when you want to upload your file just call:

new UploadFile().execute();

Edin

Hi, what is AndroidMultiPartEntity please explain... and if i want to upload pdf, word or xls file what i have to do, please give some guidance... i am new to this. – Nieberg

@amitpandya I've changed the code to a generic file upload so it's more clear to anyone reading it – Edin

Answer 8

I wanted send some strings to backend server. I didnt use json with multipart, I have used request params.

@RequestMapping(value = "/upload", method = RequestMethod.POST)
public void uploadFile(HttpServletRequest request,
        HttpServletResponse response, @RequestParam("uuid") String uuid,
        @RequestParam("type") DocType type,
        @RequestParam("file") MultipartFile uploadfile)

Url would look like

http://localhost:8080/file/upload?uuid=46f073d0&type=PASSPORT

I am passing two params (uuid and type) along with file upload. Hope this will help who don't have the complex json data to send.

Turoff

Answer 9

You could try using https://square.github.io/okhttp/ library. You can set the request body to multipart and then add the file and json objects separately like so:

MultipartBody requestBody = new MultipartBody.Builder()
                .setType(MultipartBody.FORM)
                .addFormDataPart("uploadFile", uploadFile.getName(), okhttp3.RequestBody.create(uploadFile, MediaType.parse("image/png")))
                .addFormDataPart("file metadata", json)
                .build();

        Request request = new Request.Builder()
                .url("https://uploadurl.com/uploadFile")
                .post(requestBody)
                .build();

        try (Response response = client.newCall(request).execute()) {
            if (!response.isSuccessful()) throw new IOException("Unexpected code " + response);

            logger.info(response.body().string());

Boozer

Answer 10

@RequestMapping(value = "/uploadImageJson", method = RequestMethod.POST)
    public @ResponseBody Object jsongStrImage(@RequestParam(value="image") MultipartFile image, @RequestParam String jsonStr) {
-- use  com.fasterxml.jackson.databind.ObjectMapper convert Json String to Object
}

Otherworld

Answer 11

Please ensure that you have following import. Ofcourse other standard imports

import org.springframework.core.io.FileSystemResource


    void uploadzipFiles(String token) {

        RestBuilder rest = new RestBuilder(connectTimeout:10000, readTimeout:20000)

        def zipFile = new File("testdata.zip")
        def Id = "001G00000"
        MultiValueMap<String, String> form = new LinkedMultiValueMap<String, String>()
        form.add("id", id)
        form.add('file',new FileSystemResource(zipFile))
        def urld ='''http://URL''';
        def resp = rest.post(urld) {
            header('X-Auth-Token', clientSecret)
            contentType "multipart/form-data"
            body(form)
        }
        println "resp::"+resp
        println "resp::"+resp.text
        println "resp::"+resp.headers
        println "resp::"+resp.body
        println "resp::"+resp.status
    }

Spirited

This get java.lang.ClassCastException: org.springframework.core.io.FileSystemResource cannot be cast to java.lang.String – Devoirs