Expand Authors Top

If you have a few years of experience in the Java ecosystem and you’d like to share that with the community, have a look at our Contribution Guidelines.

November Discount Launch 2022 – Top
We’re finally running a Black Friday launch. All Courses are 30% off until next Friday:

>> GET ACCESS NOW

Expanded Audience – Frontegg – Security (partner)
announcement - icon User management is very complex, when implemented properly. No surprise here.

Not having to roll all of that out manually, but instead integrating a mature, fully-fledged solution - yeah, that makes a lot of sense.
That's basically what Frontegg is - User Management for your application. It's focused on making your app scalable, secure and enjoyable for your users.
From signup to authentication, it supports simple scenarios all the way to complex and custom application logic.

Have a look:

>> Elegant User Management, Tailor-made for B2B SaaS

NPI – Lightrun – Spring (partner)

We rely on other people’s code in our own work. Every day. It might be the language you’re writing in, the framework you’re building on, or some esoteric piece of software that does one thing so well you never found the need to implement it yourself.

The problem is, of course, when things fall apart in production - debugging the implementation of a 3rd party library you have no intimate knowledge of is, to say the least, tricky. It’s difficult to understand what talks to what and, specifically, which part of the underlying library is at fault.

Lightrun is a new kind of debugger.

It's one geared specifically towards real-life production environments. Using Lightrun, you can drill down into running applications, including 3rd party dependencies, with real-time logs, snapshots, and metrics. No hotfixes, redeployments, or restarts required.

Learn more in this quick, 5-minute Lightrun tutorial:

>> The Essential List of Spring Boot Annotations and Their Use Cases

1. Overview

In this tutorial, we're going to show different techniques on how to download large files with RestTemplate.

2. RestTemplate

RestTemplate is a blocking and synchronous HTTP Client introduced in Spring 3. According to the Spring documentation, it'll be deprecated in the future since they've introduced WebClient as a reactive nonblocking HTTP client in version 5.

3. Pitfalls

Usually, when we download a file, we store it on our file system or load it into memory as a byte array. But when it's a large file, in-memory loading may lead to an OutOfMemoryError. Hence, we have to store data in a file as we read chunks of response.

Let's first look at a couple of ways that don't work:

First, what happens if we return a Resource as our return type:

Resource download() {
    return new ClassPathResource(locationForLargeFile);
}

The reason this doesn't work is that ResourceHttpMesssageConverter will load the entire response body into a ByteArrayInputStream still adding the memory pressure we wanted to avoid.

Second, what if we return an InputStreamResource and configure ResourceHttpMessageConverter#supportsReadStreaming? Well, this doesn't work either since by the time we can call  InputStreamResource.getInputStream(), we get a “socket closed” error! This is because the “execute” closes the response input stream before the exit.

So what can we do to solve the problem? Actually, there are two things here, too:

  • Write a custom HttpMessageConverter that supports File as a return type
  • Use RestTemplate.execute with a custom ResponseExtractor to store the input stream in a File

In this tutorial, we'll use the second solution because it is more flexible and also needs less effort.

4. Download Without Resume

Let's implement a ResponseExtractor to write the body to a temporary file:

File file = restTemplate.execute(FILE_URL, HttpMethod.GET, null, clientHttpResponse -> {
    File ret = File.createTempFile("download", "tmp");
    StreamUtils.copy(clientHttpResponse.getBody(), new FileOutputStream(ret));
    return ret;
});

Assert.assertNotNull(file);
Assertions
  .assertThat(file.length())
  .isEqualTo(contentLength);

Here we have used the StreamUtils.copy to copy the response input stream in a FileOutputStream, but other techniques and libraries are also available.

5. Download with Pause and Resume

As we're going to download a large file, it's reasonable to consider downloading after we've paused for some reason.

So first let's check if the download URL supports resume:

HttpHeaders headers = restTemplate.headForHeaders(FILE_URL);

Assertions
  .assertThat(headers.get("Accept-Ranges"))
  .contains("bytes");
Assertions
  .assertThat(headers.getContentLength())
  .isGreaterThan(0);

Then we can implement a RequestCallback to set “Range” header and resume the download:

restTemplate.execute(
  FILE_URL,
  HttpMethod.GET,
  clientHttpRequest -> clientHttpRequest.getHeaders().set(
    "Range",
    String.format("bytes=%d-%d", file.length(), contentLength)),
    clientHttpResponse -> {
        StreamUtils.copy(clientHttpResponse.getBody(), new FileOutputStream(file, true));
    return file;
});

Assertions
  .assertThat(file.length())
  .isLessThanOrEqualTo(contentLength);

If we don't know the exact content length, we can set the Range header value using String.format:

String.format("bytes=%d-", file.length())

6. Conclusion

We've discussed problems that can arise when downloading a large file. We also presented a solution while using RestTemplate. Finally, we've shown how we can implement a resumable download.

As always the code is available in our GitHub.

November Discount Launch 2022 – Bottom
We’re finally running a Black Friday launch. All Courses are 30% off until next Friday:

>> GET ACCESS NOW

HTTPClient footer
Comments are closed on this article!