Upload on S3 With the jclouds Library
Last updated: February 12, 2025
Mocking is an essential part of unit testing, and the Mockito library makes it easy to write clean and intuitive unit tests for your Java code.
Get started with mocking and improve your application tests using our Mockito guide:
Handling concurrency in an application can be a tricky process with many potential pitfalls. A solid grasp of the fundamentals will go a long way to help minimize these issues.
Get started with understanding multi-threaded applications with our Java Concurrency guide:
Spring 5 added support for reactive programming with the Spring WebFlux module, which has been improved upon ever since. Get started with the Reactor project basics and reactive programming in Spring Boot:
Since its introduction in Java 8, the Stream API has become a staple of Java development. The basic operations like iterating, filtering, mapping sequences of elements are deceptively simple to use.
But these can also be overused and fall into some common pitfalls.
To get a better understanding on how Streams work and how to combine them with other language features, check out our guide to Java Streams:
Explore Spring Boot 3 and Spring 6 in-depth through building a full REST API with the framework:
Yes, Spring Security can be complex, from the more advanced functionality within the Core to the deep OAuth support in the framework.
I built the security material as two full courses - Core and OAuth, to get practical with these more complex scenarios. We explore when and how to use each feature and code through it on the backing project.
You can explore the course here:
Spring Data JPA is a great way to handle the complexity of JPA with the powerful simplicity of Spring Boot.
Get started with Spring Data JPA through the guided reference course:
Refactor Java code safely — and automatically — with OpenRewrite.
Refactoring big codebases by hand is slow, risky, and easy to put off. That’s where OpenRewrite comes in. The open-source framework for large-scale, automated code transformations helps teams modernize safely and consistently.
Each month, the creators and maintainers of OpenRewrite at Moderne run live, hands-on training sessions — one for newcomers and one for experienced users. You’ll see how recipes work, how to apply them across projects, and how to modernize code with confidence.
Join the next session, bring your questions, and learn how to automate the kind of work that usually eats your sprint time.
Regression testing is an important step in the release process, to ensure that new code doesn't break the existing functionality. As the codebase evolves, we want to run these tests frequently to help catch any issues early on.
The best way to ensure these tests run frequently on an automated basis is, of course, to include them in the CI/CD pipeline. This way, the regression tests will execute automatically whenever we commit code to the repository.
In this tutorial, we'll see how to create regression tests using Selenium, and then include them in our pipeline using GitHub Actions:, to be run on the LambdaTest cloud grid:
There are several good ways to upload content to an S3 bucket in the Java world – in this article we’ll look at what the jclouds library provides for this purpose.
To use jclouds – specifically the APIs discussed in this article, this simple Maven dependency should be added to the pom of the project:
<dependency>
<groupId>org.jclouds</groupId>
<artifactId>jclouds-allblobstore</artifactId>
<version>1.5.10</version>
</dependency>1. Uploading to Amazon S3
The first step, in order to access any of these APIs, is to create a BlobStoreContext:
BlobStoreContext context =
ContextBuilder.newBuilder("aws-s3").credentials(identity, credentials)
.buildView(BlobStoreContext.class);This represents the entry-point to a general key-value storage service, such as Amazon S3 – but not limited to it.
For the more specific S3 only implementation, the context can be created similarly:
BlobStoreContext context =
ContextBuilder.newBuilder("aws-s3").credentials(identity, credentials)
.buildView(S3BlobStoreContext.class);And even more specifically:
BlobStoreContext context =
ContextBuilder.newBuilder("aws-s3").credentials(identity, credentials)
.buildView(AWSS3BlobStoreContext.class);When the authenticated context is no longer needed, closing it is required to release all resources – threads and connections – associated to it.
2. The Four S3 APIs of jclouds
The jclouds library provides four different APIs to upload content to S3 bucket, ranging from simple but inflexible to complex and powerful, all obtained via the BlobStoreContext. Let’s start with the simplest.
2.1. Upload via the Map API
The easiest way jclouds can be used to interact with an S3 bucket is by representing that bucket as a Map. The API is obtained from the context:
InputStreamMap bucket = context.createInputStreamMap("bucketName");Then, to upload a simple HTML file:
bucket.putString("index1.html", "<html><body>hello world1</body></html>");The InputStreamMap API exposes several other types of PUT operations – files, raw bytes – both for single and bulk.
A simple integration test can be used as an example:
@Test
public void whenFileIsUploadedToS3WithMapApi_thenNoExceptions() {
BlobStoreContext context =
ContextBuilder.newBuilder("aws-s3").credentials(identity, credentials)
.buildView(AWSS3BlobStoreContext.class);
InputStreamMap bucket = context.createInputStreamMap("bucketName");
bucket.putString("index1.html", "<html><body>hello world1</body></html>");
context.close();
}2.2. Upload via BlobMap
Using the simple Map API is straightforward but ultimately limited – for example, there is no way to pass in metadata about the content being uploaded. When more flexibility and customization is necessary, this simplified approach to uploading data to S3 via a Map is no longer enough.
The next API we’ll look at is the Blob Map API – this is obtained from the context:
BlobMap bucket = context.createBlobMap("bucketName");The API allows the client to access more lower level details, such as Content–Length, Content-Type, Content-Encoding, eTag hash and others; to upload new content in the bucket:
Blob blob = bucket.blobBuilder().name("index2.html").
payload("<html><body>hello world2</body></html>").
contentType("text/html").calculateMD5().build();The API also allows setting a variety of payloads on the create request.
A simple integration test for uploading a basic HTML file to S3 via the Blob Map API:
@Test
public void whenFileIsUploadedToS3WithBlobMap_thenNoExceptions() throws IOException {
BlobStoreContext context =
ContextBuilder.newBuilder("aws-s3").credentials(identity, credentials)
.buildView(AWSS3BlobStoreContext.class);
BlobMap bucket = context.createBlobMap("bucketName");
Blob blob = bucket.blobBuilder().name("index2.html").
payload("<html><body>hello world2</body></html>").
contentType("text/html").calculateMD5().build();
bucket.put(blob.getMetadata().getName(), blob);
context.close();
}2.3. Upload via BlobStore
The previous APIs had no way to upload content using multipart upload – this makes them ill suited when working with large files. This limitation is addressed by the next API we’re going to look at – the synchronous BlobStore API.
This is obtained from the context:
BlobStore blobStore = context.getBlobStore();To use the multipart support and upload a file to S3:
Blob blob = blobStore.blobBuilder("index3.html").
payload("<html><body>hello world3</body></html>").contentType("text/html").build();
blobStore.putBlob("bucketName", blob, PutOptions.Builder.multipart());The payload builder is the same one that was being used by the BlobMap API, so the same flexibility in specifying lower level metadata information about the blob is available here. The difference is the PutOptions supported by the PUT operation of the API – namely the multipart support.
The previous integration test now has multipart enabled:
@Test
public void whenFileIsUploadedToS3WithBlobStore_thenNoExceptions() {
BlobStoreContext context =
ContextBuilder.newBuilder("aws-s3").credentials(identity, credentials)
.buildView(AWSS3BlobStoreContext.class);
BlobStore blobStore = context.getBlobStore();
Blob blob = blobStore.blobBuilder("index3.html").
payload("<html><body>hello world3</body></html>").contentType("text/html").build();
blobStore.putBlob("bucketName", blob, PutOptions.Builder.multipart());
context.close();
}2.4. Upload via AsyncBlobStore
While the previous BlobStore API was synchronous, there is also an asynchronous API for BlobStore – AsyncBlobStore. The API is similarly obtained from the context:
AsyncBlobStore blobStore = context.getAsyncBlobStore();The only difference between the two is that the async API is returning ListenableFuture for the PUT asynchronous operation:
Blob blob = blobStore.blobBuilder("index4.html").
.payload("<html><body>hello world4</body></html>").build();
blobStore.putBlob("bucketName", blob).get();The integration test displaying this operation is similar to the synchronous one:
@Test
public void whenFileIsUploadedToS3WithBlobStore_thenNoExceptions() {
BlobStoreContext context =
ContextBuilder.newBuilder("aws-s3").credentials(identity, credentials)
.buildView(AWSS3BlobStoreContext.class);
BlobStore blobStore = context.getBlobStore();
Blob blob = blobStore.blobBuilder("index4.html").
payload("<html><body>hello world4</body></html>").contentType("text/html").build();
Future<String> putOp = blobStore.putBlob("bucketName", blob, PutOptions.Builder.multipart());
putOp.get();
context.close();
}3. Conclusion
In this article, we analyzed the four APIs that the jclouds library provides to upload content to Amazon S3. These four APIs are generic and they work with other key-value storage services as well – such as Microsoft Azure Storage for example.
In the next article we’ll look at the Amazon specific S3 API available in jclouds – the AWSS3Client. We’ll implement the operation of uploading a large file, dynamically calculate the optimal number of parts for any given file, and perform the upload of all parts in parallel.
