How to forward large files with RestTemplate?

Question

I have a web service call through which zip files can be uploaded. The files are then forwarded to another service for storage, unzipping, etc. For now the file is stored on the file system, then a FileSystemResource is built.

Resource zipFile = new FileSystemResource(tempFile.getAbsolutePath());

I could use a ByteStreamResource in order to save time(the saving of the file on disk is not needed before forwarding) but for that i need to build a byte array. In case of large files I will get an "OutOfMemory : java heap space" error.

ByteArrayResource r = new ByteArrayResource(inputStream.getBytes());

Any solutions to forwarding files without getting an OutOfMemory error using RestTemplate?

Richellericher

Can't you pass the inputstream to the other service? Or you'll have to write the inputstream to a file and then pass the file handle to the service. Also, not sure how this relates to Groovy? – Bocock

I didn't find any way to just pass the input stream. I used the Groovy tag because the code is in groovy (java InputStream does not have a getBytes method) – Richellericher

Ahhh, I was thrown as you're writing it in a very Java style ;-) SO what does this other service accept then? – Bocock

I'm not providing this as an answer, cause it's a bit larger in scope than an answer to your question, but have you considered Spring Integration for this problem? You're basically looking at a Claim Check pattern, with web service and REST adapters. You can get a lot of the work done for you by the SI framework. static.springsource.org/spring-integration/reference/htmlsingle/… – Broomfield

Answer 1

Edit: The other answers are better (use Resource) https://mcmap.net/q/375915/-how-to-forward-large-files-with-resttemplate

My original answer:

You can use execute for this kind of low-level operation. In this snippet I've used Commons IO's copy method to copy the input stream. You would need to customize the HttpMessageConverterExtractor for the kind of response you're expecting.

final InputStream fis = new FileInputStream(new File("c:\\autoexec.bat")); // or whatever
final RequestCallback requestCallback = new RequestCallback() {
     @Override
    public void doWithRequest(final ClientHttpRequest request) throws IOException {
        request.getHeaders().add("Content-type", "application/octet-stream");
        IOUtils.copy(fis, request.getBody());
     }
};
final RestTemplate restTemplate = new RestTemplate();
SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);     
restTemplate.setRequestFactory(requestFactory);     
final HttpMessageConverterExtractor<String> responseExtractor =
    new HttpMessageConverterExtractor<String>(String.class, restTemplate.getMessageConverters());
restTemplate.execute("http://localhost:4000", HttpMethod.POST, requestCallback, responseExtractor);

(Thanks to Baz for pointing out you need to call setBufferRequestBody(false) or it will defeat the point)

Placid

Isn't the input stream all loaded into memory this way ? – Gadfly

@Gadfly No, it'll be loaded piecemeal during the copying. Should be quite memory-efficient. – Placid

doWithRequest(request) is called before request.execute() and it will not return until all the stream is copied to the request body. The cintent is copied in chunks to the body but nobody consumes from it, it does not act like a pipe, am I missing smth ? – Gadfly

@Gadfly doWithRequest is in an anonymous class; it's defined before, but won't be called until, execute (it's a callback) – Placid

I'm aware of Java semantics :) I talk about the RestTemplate class, it execute the instance of callback you pass, then call the execute method – Gadfly

@Gadfly you may have a point... I've tested this and the request sets the Content-length header, which it could only do by buffering the entire contents. Will update the answer when I figure it out. – Placid

found kind of solution, using *ClientHttpRequestFactory#setBufferRequestBody(false) generate in turn a *StreamingClientHttpRequest which open the connection when getBody() is called. – Gadfly

can't we just use new ByteArrayResource(inputStream.getBytes()) with ClientHttpRequestFactory#setBufferRequestBody(false) set on the RestTemplate? and then just call RestTemplate#exchange()? – Apus

@MaxCh there is no getBytes method on InputStream – Placid

This is right, the original question was written in Groovy, so I just followed that. But my point is that using execute may not be necessary with setBufferRequestBody(false) called. So code can be simpler. Right? – Apus

@Placid I also have similar question here on RestTemplate in which I am having doubts on ClientHttpRequestFactory implementations. See if you can help me out if possible. I am stuck on this for a while. Any help will be appreciated. – Erme

Answer 2

The only part of @artbristol's answer you really need is this (which you can set up as a RestTemplate Spring bean):

final RestTemplate restTemplate = new RestTemplate();
SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);     
restTemplate.setRequestFactory(requestFactory);     

After that, I think just using a FileSystemResource as your request body will do the right thing.

I've also used an InputStreamResource successfully this way, for cases where you already have the data as an InputStream and don't need to consume it multiple times.

In my case, we had gzipped our files and wrapped a GZipInputStream in an InputStreamResource.

Seale

This solved my original problem, and the rest template was able to process the large file without running out of memory thanks a lot! After that I started getting Error writing body to server and I was able to solve this one with the response in this post. I'm leaving this here in case anyone else face the same problem. Thanks! – Majuscule

Answer 3

I think that the above answer has unnecessary code - you don't need to make an anonymous RequestCallback inner class, and you don't need to use IOUtils from apache.

I spent a bit of time researching a similar solution to yours and this is what I came up with:

You can accomplish your goal much simpler by using the Spring Resource Interface and RestTemplate.

RestTemplate restTemplate = new RestTemplate();

SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);
restTemplate.setRequestFactory(requestFactory);

File file = new File("/whatever");

HttpEntity<FileSystemResource> requestEntity = new HttpEntity<>(new FileSystemResource(file));
ResponseEntity e = restTemplate.exchange("http://localhost:4000", HttpMethod.POST, requestEntity, Map.class);

(This example assumes that the response from where you are POSTing to is JSON. But, this can easily be changed by changing the return type class... set to Map.class above)

Undercurrent

Answer 4

One additional thing to note when doing large uploads is presence of Spring Actuator. That automatically injects MetricsRestTemplateCustomizer which will cause your request to be buffered, even if you set setBufferRequestBody(false) on your factory.

Two ways to get around it, probably more options:

1. Disable the customizer globally -

@SpringBootApplication(exclude = HttpClientMetricsAutoConfiguration.class)

2. Disable for only some of the rest templates, if you're building a restTemplate via restTemplateBuilder.

            var restTemplate = restTemplateBuilder.requestFactory(() ->
                    {
                        var reqFact = new HttpComponentsClientHttpRequestFactory(HttpClientBuilder.create()//
                                .setDefaultRequestConfig(RequestConfig.custom().build())//
                                .build());
                        reqFact.setBufferRequestBody(false);
                        return reqFact;
                    })//
                    .setBufferRequestBody(false)//
                    //these are to disable any possible interceptor/customizer that might force buffering on the request
                    //like the MetricsRestTemplateCustomizer
                    .interceptors(List.of())//
                    .customizers(List.of())//
                    .build();

Constrain