Spring WebClient: How to stream large byte[] to file?

Question

It seems like it the Spring RestTemplate isn't able to stream a response directly to file without buffering it all in memory. What is the proper to achieve this using the newer Spring 5 WebClient?

WebClient client = WebClient.create("https://example.com");
client.get().uri(".../{name}", name).accept(MediaType.APPLICATION_OCTET_STREAM)
                    ....?

I see people have found a few workarounds/hacks to this issue with RestTemplate, but I am more interested in doing it the proper way with the WebClient.

There are many examples of using RestTemplate to download binary data but almost all of them load the byte[] into memory.

Elecampane

You can check already available #32988870 – Spirituous

Thanks but that doesn’t show how to do it using WebClient. – Elecampane

To solve the problem, you can use RestTemplate of Spring. However Spring 5 introduced Webclient. – Spirituous

You can also refer this link #46740500 – Spirituous

I don’t think it answers the question. Please create an answer if you think it does. – Elecampane

Possible duplicate of Spring WebFlux Webclient receiving an application/octet-stream file as a Mono – Homs

@Homs - Do you really think this is a duplicate of that question? For one thing that question doesn't mention streaming directly to the file (not keeping the whole response in memory), which is the main point of my question; and also that question is using Kotlin, not Java. – Elecampane

@DaveL. - Yea, you're right, should have flagged it as off topic. Let us know when you have something that's giving you problems. – Homs

> Yea, you're right, should have flagged it as off topic. @Homs I'm not sure why you keep trying to find a way to undermine my question, but feel free to review stackoverflow.com/help/on-topic and the code of conduct. – Elecampane

It's more of a request to write the code for you than to answer a question about a problem you are having. No matter, maybe someone here will do it for you. I'm not interested enough to do it myself. Later I saw answers closer than the one I posted but I didn't see anything obvious that included saving the stream as it was incoming. Seems like you'd have to open a file stream as well as the response stream and copy blocks of data between the two. – Homs

Just to clarify for others; that's not really accurate. A link to a specific example, a description + link of the correct api, or, at most, a couple lines of sample code is totally sufficient. – Elecampane

Any luck with this @DaveL ?? – Totaquine

@JamesGawron No I haven't had a chance to verify the answer below. – Elecampane

@DaveL. Any below solutions worked for you , without loading file in memory . Actually , Im also facing same problem like you . – Finespun

Answer 1

With recent stable Spring WebFlux (5.2.4.RELEASE as of writing):

final WebClient client = WebClient.create("https://example.com");
final Flux<DataBuffer> dataBufferFlux = client.get()
        .accept(MediaType.TEXT_HTML)
        .retrieve()
        .bodyToFlux(DataBuffer.class); // the magic happens here

final Path path = FileSystems.getDefault().getPath("target/example.html");
DataBufferUtils
        .write(dataBufferFlux, path, CREATE_NEW)
        .block(); // only block here if the rest of your code is synchronous

For me the non-obvious part was the bodyToFlux(DataBuffer.class), as it is currently mentioned within a generic section about streaming of Spring's documentation, there is no direct reference to it in the WebClient section.

Mckale

DataBuffer basically looks just like a ByteBuffer. I don't see any coordination happening between reader & writer, nor any way to set a size limit on the buffer. How do you know that DataBuffer is reactive (or multi-thread coordinated) & size-bounded? – Name

Never mind, apparently the stream generates as many DataBuffers as it needs to, each one containing a chunk of the response data. Not sure how the size is determined, maybe in the Netty config. And each DataBuffer is complete when it's emitted to the Flux so no coordination is needed. – Name

Thanks a lot !!! i've wasted hours to understand how to make it works :) – Pea

Answer 2

I cannot test whether or not the following code effectively does not buffer the contents of webClient payload in memory. Nevertheless, i think you should start from there:

public Mono<Void> testWebClientStreaming() throws IOException {
    Flux<DataBuffer> stream = 
            webClient
                    .get().accept(MediaType.APPLICATION_OCTET_STREAM)
                    .retrieve()
            .bodyToFlux(DataBuffer.class);
    Path filePath = Paths.get("filename");
    AsynchronousFileChannel asynchronousFileChannel = AsynchronousFileChannel.open(filePath, WRITE);
    return DataBufferUtils.write(stream, asynchronousFileChannel)
            .doOnNext(DataBufferUtils.releaseConsumer())
            .doAfterTerminate(() -> {
                try {
                    asynchronousFileChannel.close();
                } catch (IOException ignored) { }
            }).then();
}

Achromat

FYI, There is a new kid on the block. See DataBufferUtils#write(Publisher<DataBuffer>, Path, OpenOption...) – Crinkleroot

Answer 3

Store the body to a temporary file and consume

static <R> Mono<R> writeBodyToTempFileAndApply(
        final WebClient.ResponseSpec spec,
        final Function<? super Path, ? extends R> function) {
    return using(
            () -> createTempFile(null, null),
            t -> write(spec.bodyToFlux(DataBuffer.class), t)
                    .thenReturn(function.apply(t)),
            t -> {
                try {
                    deleteIfExists(t);
                } catch (final IOException ioe) {
                    throw new RuntimeException(ioe);
                }
            }
    );
}

Pipe the body and consume

static <R> Mono<R> pipeBodyAndApply(
        final WebClient.ResponseSpec spec, final ExecutorService executor,
        final Function<? super ReadableByteChannel, ? extends R> function) {
    return using(
            Pipe::open,
            p -> {
                final Future<Disposable> future = executor.submit(
                        () -> write(spec.bodyToFlux(DataBuffer.class), p.sink())
                                .log()
                                .doFinally(s -> {
                                    try {
                                        p.sink().close();
                                        log.debug("p.sink closed");
                                    } catch (final IOException ioe) {
                                        throw new RuntimeException(ioe);
                                    }
                                })
                                .subscribe(DataBufferUtils.releaseConsumer())
                );
                return just(function.apply(p.source()))
                        .log()
                        .doFinally(s -> {
                            try {
                                final Disposable disposable = future.get();
                                assert disposable.isDisposed();
                            } catch (InterruptedException | ExecutionException e) {
                                e.printStackTrace();
                            }
                        });
            },
            p -> {
                try {
                    p.source().close();
                    log.debug("p.source closed");
                } catch (final IOException ioe) {
                    throw new RuntimeException(ioe);
                }
            }
    );
}

Crinkleroot

Isn't this answer a duplicate of https://mcmap.net/q/346602/-how-to-correctly-read-flux-lt-databuffer-gt-and-convert-it-to-a-single-inputstream? – Aslam

@AbhijitSarkar Yes. – Crinkleroot

Answer 4

I'm not sure if you have access to RestTemplate in your current usage of spring, but this one have worked for me.

RestTemplate restTemplate // = ...;

RequestCallback requestCallback = request -> request.getHeaders()
        .setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));

// Streams the response
ResponseExtractor<Void> responseExtractor = response -> {
    // Here I write the response to a file but do what you like
    Path path = Paths.get("http://some/path");
    Files.copy(response.getBody(), path);
    return null;
};
restTemplate.execute(URI.create("www.something.com"), HttpMethod.GET, requestCallback, responseExtractor);

Korte

Thanks for the response. I see the code is the same as from here: https://mcmap.net/q/450498/-download-large-file-from-server-using-rest-template-java-spring-mvc. However, that doesn't work with Spring 5 or newer anymore -- see this issue: github.com/spring-projects/spring-framework/issues/19448 – Elecampane

Original poster said he'd want to do it in the proper way, with Webclient that's non blocking. – Mckale

RestTemplate is soon going to be deprecated. – Subscapular