what's the correct way to send a file from REST web service to client?

Question

I've just started to develop REST services, but I've come across a difficult situation: sending files from my REST service to my client. So far I've gotten the hang of how to send simple data types (strings, integers, etc) but sending a file is a different matter since there are so many file formats that I don't know where I should even begin. My REST service is made on Java and I'm using Jersey, I'm sending all the data using the JSON format.

I've read about base64 encoding, some people say it's a good technique, others say it isn't because of file size issues. What is the correct way? This is how a simple resource class in my project is looking:

import java.sql.SQLException;
import java.util.List;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Request;
import javax.ws.rs.core.UriInfo;

import com.mx.ipn.escom.testerRest.dao.TemaDao;
import com.mx.ipn.escom.testerRest.modelo.Tema;

@Path("/temas")
public class TemaResource {

    @GET
    @Produces({MediaType.APPLICATION_JSON})
    public List<Tema> getTemas() throws SQLException{

        TemaDao temaDao = new TemaDao();        
        List<Tema> temas=temaDao.getTemas();
        temaDao.terminarSesion();

        return temas;
    }
}

I'm guessing the code for sending a file would be something like:

import java.sql.SQLException;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;

@Path("/resourceFiles")
public class FileResource {

    @GET
    @Produces({application/x-octet-stream})
    public File getFiles() throws SQLException{ //I'm not really sure what kind of data type I should return

        // Code for encoding the file or just send it in a data stream, I really don't know what should be done here

        return file;
    }
}

What kind of annotations should I use? I've seen some people recommend for a @GET using @Produces({application/x-octet-stream}), is that the correct way? The files I'm sending are specific ones so the client doesn't need to browse through the files. Can anyone guide me into how am I supposed to send the file? Should I encode it using base64 to send it as a JSON object? or the encoding isn't necessary to send it as a JSON object? Thanks for any help you may give.

Monogamist

Do you have an actual java.io.File (or file path) on your server or is the data coming from some other source, like a database, web service, method call returning an InputStream? – Prothesis

Answer 1

I don't recommend encoding binary data in base64 and wrapping it in JSON. It will just needlessly increase the size of the response and slow things down.

Simply serve your file data using GET and application/octect-streamusing one of the factory methods of javax.ws.rs.core.Response (part of the JAX-RS API, so you're not locked into Jersey):

@GET
@Produces(MediaType.APPLICATION_OCTET_STREAM)
public Response getFile() {
  File file = ... // Initialize this to the File path you want to serve.
  return Response.ok(file, MediaType.APPLICATION_OCTET_STREAM)
      .header("Content-Disposition", "attachment; filename=\"" + file.getName() + "\"" ) //optional
      .build();
}

If you don't have an actual File object, but an InputStream, Response.ok(entity, mediaType) should be able to handle that as well.

Prothesis

thanks, this worked great, but what if I want to consume a whole folder structure? I was thinking something like this Also since I'll be receiving various files on the client, how should I treat the HttpResponse's entity response? – Monogamist

Take a look at ZipOutputStream along with returning a StreamingOutput from getFile(). This way you get a well-known multi-file format that most clients should easily be able to read. Use compression only if it makes sense for your data, i.e. not for pre-compressed files like JPEGs. On the client side, there's ZipInputStream to parse the response. – Prothesis

This might help: #10101436 – Weihs

Is there a way to add metadata of the file in the response along with file binary data? – Maidenly

You can always add more headers to the response. If that's not enough you'll have to encode it into the octet stream, i.e. serve a container format that has both metadata and the file you want. – Prothesis

It works also with a byte array, see here: #3496709 – Overscore

I got the response as below.where is the actual file.my file contains json data. Does the android application on the other end wil get a file. ?? please suggest solutions. { "statusType": "OK", "entity": ** "entityType": "java.lang.String", "metadata": { "Content-Disposition": [ "attachment; filename="/home/cine/WORKSPACES/study/.metadata/.plugins/org.eclipse.wst.server.core/tmp0/wtpwebapps/Cine Sat May 16 12:05:07 IST 2015.txt"" ] }, "status": 200 } – Shortwinded

@Shortwinded Please open a new question, I suspect this is related to the Content-Disposition header that user bluish added to my original answer. – Prothesis

@Philipp Reichart here is the link. #30296857 – Shortwinded

@PhilippReichart- can you please share an example of sending ZipOutputStream from rest service – Copro

@PhilippReichart Can you confirm if i call this getFile endpoint(of server-1) from another rest service(on server-2), will whole inputStream content be first loaded into memory then returned as response or inputStream will be returned to server-2 in some sense? If it's later, that means we're serialising and deserialising streams?? will the http connection be kept open until client fully go over stream and read all content? If its former, isn't it slow down server-1 in case of loading larger chunk of data into memory. Let me know if its not the right way to ask, i'll create a new question – Unfathomable

@DeepankarSingh I haven't looked at the JAX-RS spec or any code, but I'm guessing the response handler will just stream the File/other input straight into the HttpServletOutputStream. There is no reason to load it into memory. There is no Java serialization of Java InputStreams going on, server-1 and server-2 just communicate over HTTP, and the file you're sending is the payload of the HTTP response from server-1 to server-2. The HTTP connection will be kept open until server-2 has read the response, server-2 closed the connection (discarding the response) or the connection times out. – Prothesis

Answer 2

If you want to return a File to be downloaded, specially if you want to integrate with some javascript libs of file upload/download, then the code bellow should do the job:

@GET
@Path("/{key}")
public Response download(@PathParam("key") String key,
                         @Context HttpServletResponse response) throws IOException {
    try {
        //Get your File or Object from wherever you want...
            //you can use the key parameter to indentify your file
            //otherwise it can be removed
        //let's say your file is called "object"
        response.setContentLength((int) object.getContentLength());
        response.setHeader("Content-Disposition", "attachment; filename="
                + object.getName());
        ServletOutputStream outStream = response.getOutputStream();
        byte[] bbuf = new byte[(int) object.getContentLength() + 1024];
        DataInputStream in = new DataInputStream(
                object.getDataInputStream());
        int length = 0;
        while ((in != null) && ((length = in.read(bbuf)) != -1)) {
            outStream.write(bbuf, 0, length);
        }
        in.close();
        outStream.flush();
    } catch (S3ServiceException e) {
        e.printStackTrace();
    } catch (ServiceException e) {
        e.printStackTrace();
    }
    return Response.ok().build();
}

Whacky

Answer 3

Change the machine address from localhost to IP address you want your client to connect with to call below mentioned service.

Client to call REST webservice:

package in.india.client.downloadfiledemo;

import java.io.BufferedInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;

import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response.Status;

import com.sun.jersey.api.client.Client;
import com.sun.jersey.api.client.ClientHandlerException;
import com.sun.jersey.api.client.ClientResponse;
import com.sun.jersey.api.client.UniformInterfaceException;
import com.sun.jersey.api.client.WebResource;
import com.sun.jersey.multipart.BodyPart;
import com.sun.jersey.multipart.MultiPart;

public class DownloadFileClient {

    private static final String BASE_URI = "http://localhost:8080/DownloadFileDemo/services/downloadfile";

    public DownloadFileClient() {

        try {
            Client client = Client.create();
            WebResource objWebResource = client.resource(BASE_URI);
            ClientResponse response = objWebResource.path("/")
                    .type(MediaType.TEXT_HTML).get(ClientResponse.class);

            System.out.println("response : " + response);
            if (response.getStatus() == Status.OK.getStatusCode()
                    && response.hasEntity()) {
                MultiPart objMultiPart = response.getEntity(MultiPart.class);
                java.util.List<BodyPart> listBodyPart = objMultiPart
                        .getBodyParts();
                BodyPart filenameBodyPart = listBodyPart.get(0);
                BodyPart fileLengthBodyPart = listBodyPart.get(1);
                BodyPart fileBodyPart = listBodyPart.get(2);

                String filename = filenameBodyPart.getEntityAs(String.class);
                String fileLength = fileLengthBodyPart
                        .getEntityAs(String.class);
                File streamedFile = fileBodyPart.getEntityAs(File.class);

                BufferedInputStream objBufferedInputStream = new BufferedInputStream(
                        new FileInputStream(streamedFile));

                byte[] bytes = new byte[objBufferedInputStream.available()];

                objBufferedInputStream.read(bytes);

                String outFileName = "D:/"
                        + filename;
                System.out.println("File name is : " + filename
                        + " and length is : " + fileLength);
                FileOutputStream objFileOutputStream = new FileOutputStream(
                        outFileName);
                objFileOutputStream.write(bytes);
                objFileOutputStream.close();
                objBufferedInputStream.close();
                File receivedFile = new File(outFileName);
                System.out.print("Is the file size is same? :\t");
                System.out.println(Long.parseLong(fileLength) == receivedFile
                        .length());
            }
        } catch (UniformInterfaceException e) {
            e.printStackTrace();
        } catch (ClientHandlerException e) {
            e.printStackTrace();
        } catch (FileNotFoundException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }

    }

    public static void main(String... args) {
        new DownloadFileClient();
    }
}

Service to response client:

package in.india.service.downloadfiledemo;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;

import com.sun.jersey.multipart.MultiPart;

@Path("downloadfile")
@Produces("multipart/mixed")
public class DownloadFileResource {

    @GET
    public Response getFile() {

        java.io.File objFile = new java.io.File(
                "D:/DanGilbert_2004-480p-en.mp4");
        MultiPart objMultiPart = new MultiPart();
        objMultiPart.type(new MediaType("multipart", "mixed"));
        objMultiPart
                .bodyPart(objFile.getName(), new MediaType("text", "plain"));
        objMultiPart.bodyPart("" + objFile.length(), new MediaType("text",
                "plain"));
        objMultiPart.bodyPart(objFile, new MediaType("multipart", "mixed"));

        return Response.ok(objMultiPart).build();

    }
}

JAR needed:

jersey-bundle-1.14.jar
jersey-multipart-1.14.jar
mimepull.jar

WEB.XML:

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns="http://java.sun.com/xml/ns/javaee" xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
    xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
    id="WebApp_ID" version="2.5">
    <display-name>DownloadFileDemo</display-name>
    <servlet>
        <display-name>JAX-RS REST Servlet</display-name>
        <servlet-name>JAX-RS REST Servlet</servlet-name>
        <servlet-class>com.sun.jersey.spi.container.servlet.ServletContainer</servlet-class>
        <init-param>
             <param-name>com.sun.jersey.config.property.packages</param-name> 
             <param-value>in.india.service.downloadfiledemo</param-value>
        </init-param>
        <load-on-startup>1</load-on-startup>
    </servlet>
    <servlet-mapping>
        <servlet-name>JAX-RS REST Servlet</servlet-name>
        <url-pattern>/services/*</url-pattern>
    </servlet-mapping>
    <welcome-file-list>
        <welcome-file>index.jsp</welcome-file>
    </welcome-file-list>
</web-app>

Flinch

Answer 4

Since youre using JSON, I would Base64 Encode it before sending it across the wire.

If the files are large, try to look at BSON, or some other format that is better with binary transfers.

You could also zip the files, if they compress well, before base64 encoding them.

Lovelace

I was planning to zip them before sending them for the whole file size reason, but if I base64 encode it, what should my @Produces annotation contain? – Monogamist

application/json as per JSON spec, regardless of what you put into it. (ietf.org/rfc/rfc4627.txt?number=4627) Bear in mind the base64 encoded file should still be inside JSON tags – Lovelace

There's no benefit in encoding binary data in base64 and then wrapping it in JSON. It will just needlessly increase the size of the response and slow things down. – Prothesis