Intro to Google Cloud Storage with Java

1. Overview

Google Cloud Storage offers online storage tailored to an individual application’s needs based on location, the frequency of access, and cost. Unlike Amazon Web Services, Google Cloud Storage uses a single API for high, medium, and low-frequency access.

Like most cloud platforms, Google offers a free tier of access; the pricing details are here.

In this tutorial, we’ll connect to storage, create a bucket, write, read, and update data. While using the API to read and write data, we’ll also use the gsutil cloud storage utility.

2. Google Cloud Storage Setup

2.1. Maven Dependency

We need to add a single dependency to our pom.xml:

<dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud-storage</artifactId>
    <version>1.17.0</version>
</dependency>

Maven Central has the latest version of the library.

2.2. Create Authentication Key

Before we can connect to Google Cloud, we need to configure authentication. Google Cloud Platform (GCP) applications load a private key and configuration information from a JSON configuration file. We generate this file via the GCP console. Access to the console requires a valid Google Cloud Platform Account.

We create our configuration by:

1. Going to the Google Cloud Platform Console
2. If we haven’t yet defined a GCP project, we click the create button and enter a project name, such as “baeldung-cloud-tutorial“
3. Select “new service account” from the drop-down list
4. Add a name such as “baeldung-cloud-storage” into the account name field.
5. Under “role” select Project, and then Owner in the submenu.
6. Select create, and the console downloads a private key file.

The role in step #6 authorizes the account to access project resources. For the sake of simplicity, we gave this account complete access to all project resources.

For a production environment, we would define a role that corresponds to the access the application needs.

2.3. Install the Authentication Key

Next, we copy the file downloaded from GCP console to a convenient location and point the GOOGLE_APPLICATION_CREDENTIALS environment variable at it. This is the easiest way to load the credentials, although we’ll look at another possibility below.

For Linux or Mac:

export GOOGLE_APPLICATION_CREDENTIALS="/path/to/file"

For Windows:

set GOOGLE_APPLICATION_CREDENTIALS="C:\path\to\file"

2.4. Install Cloud Tools

Google provides several tools for managing their cloud platform. We’re going to use gsutil during this tutorial to read and write data alongside the API.

We can do this in two easy steps:

1. Install the Cloud SDK from the instructions here for our platform.
2. Follow the Quickstart for our platform here. In step 4 of Initialize the SDK, we select the project name in step 4 of section 2.2 above (“baeldung-cloud-storage” or whichever name you used).

gsutil is now installed and configured to read data from our cloud project.

3. Connecting to Storage and Creating a Bucket

3.1. Connect to Storage

Before we can use Google Cloud storage, we have to create a service object. If we’ve already set up the GOOGLE_APPLICATION_CREDENTIALS environment variable, we can use the default instance:

Storage storage = StorageOptions.getDefaultInstance().getService();

If we don’t want to use the environment variable, we have to create a Credentials instance and pass it to Storage with the project name:

Credentials credentials = GoogleCredentials
  .fromStream(new FileInputStream("path/to/file"));
Storage storage = StorageOptions.newBuilder().setCredentials(credentials)
  .setProjectId("baeldung-cloud-tutorial").build().getService();

3.2. Creating a Bucket

Now that we’re connected and authenticated, we can create a bucket. Buckets are containers that hold objects. They can be used to organize and control data access.

There is no limit to the number of objects in a bucket. GCP limits the numbers of operations on buckets and encourages application designers to emphasize operations on objects rather than on buckets.

Creating a bucket requires a BucketInfo:

Bucket bucket = storage.create(BucketInfo.of("baeldung-bucket"));

For this simple example, we a bucket name and accept the default properties. Bucket names must be globally unique. If we choose a name that is already used, create() will fail.

3.3. Examing a Bucket With gsutil

Since we have a bucket now, we can take a examine it with gsutil.

Let’s open a command prompt and take a look:

$ gsutil ls -L -b gs://baeldung-1-bucket/
gs://baeldung-1-bucket/ :
	Storage class:			STANDARD
	Location constraint:		US
	Versioning enabled:		None
	Logging configuration:		None
	Website configuration:		None
	CORS configuration: 		None
	Lifecycle configuration:	None
	Requester Pays enabled:		None
	Labels:				None
	Time created:			Sun, 11 Feb 2018 21:09:15 GMT
	Time updated:			Sun, 11 Feb 2018 21:09:15 GMT
	Metageneration:			1
	ACL:
	  [
	    {
	      "entity": "project-owners-385323156907",
	      "projectTeam": {
	        "projectNumber": "385323156907",
	        "team": "owners"
	      },
	      "role": "OWNER"
	    },
	    ...
	  ]
	Default ACL:
	  [
	    {
	      "entity": "project-owners-385323156907",
	      "projectTeam": {
	        "projectNumber": "385323156907",
	        "team": "owners"
	      },
	      "role": "OWNER"
	    },
            ...
	  ]

gsutil looks a lot like shell commands, and anyone familiar with the Unix command line should feel very comfortable here. Notice we passed in the path to our bucket as a URL: gs://baeldung-1-bucket/, along with a few other options.

The ls option produces a listing or objects or buckets, and the -L option indicated that we want a detailed listing – so we received details about the bucket including the creation times and access controls.

Let’s add some data to our bucket!

4. Reading, Writing and Updating Data

In Google Cloud Storage, objects are stored in Blobs; Blob names can contain any Unicode character, limited to 1024 characters.

4.1. Writing Data

Let’s save a String to our bucket:

String value = "Hello, World!";
byte[] bytes = value.getBytes(UTF_8);
Blob blob = bucket.create("my-first-blob", bytes);

As you can see, objects are simply arrays of bytes in the bucket, so we store a String by simply operating with its raw bytes.

4.2. Reading Data with gsutil

Now that we have a bucket with an object in it, let’s take a look at gsutil.

Let’s start by listing the contents of our bucket:

$ gsutil ls gs://baeldung-1-bucket/
gs://baeldung-1-bucket/my-first-blob

We passed gsutil the ls option again but omitted -b and -L, so we asked for a brief listing of objects. We receive a list of URIs for each object, which is one in our case.

Let’s examine the object:

$ gsutil cat gs://baeldung-1-bucket/my-first-blob
Hello World!

Cat concatenates the contents of the object to standard output. We see the String we wrote to the Blob.

4.3. Reading Data

Blobs are assigned a BlobId upon creation.

The easiest way to retrieve a Blob is with the BlobId:

Blob blob = storage.get(blobId);
String value = new String(blob.getContent());

We pass the id to Storage and get the Blob in return, and getContent() returns the bytes.

If we don’t have the BlobId, we can search the Bucket by name:

Page<Blob> blobs = bucket.list();
for (Blob blob: blobs.getValues()) {
    if (name.equals(blob.getName())) {
        return new String(blob.getContent());
    }
}

4.4. Updating Data

We can update a Blob by retrieving it and then accessing its WriteableByteChannel:

String newString = "Bye now!";
Blob blob = storage.get(blobId);
WritableByteChannel channel = blob.writer();
channel.write(ByteBuffer.wrap(newString.getBytes(UTF_8)));
channel.close();

Let’s examine the updated object:

$ gsutil cat gs://baeldung-1-bucket/my-first-blob
Bye now!

4.5. Save an Object to File, Then Delete

Let’s save the updated the object to a file:

$ gsutil copy gs://baeldung-1-bucket/my-first-blob my-first-blob
Copying gs://baeldung-1-bucket/my-first-blob...
/ [1 files][    9.0 B/    9.0 B]
Operation completed over 1 objects/9.0 B.
Grovers-Mill:~ egoebelbecker$ cat my-first-blob
Bye now!

As expected, the copy option copies the object to the filename specified on the command line.

gsutil can copy any object from Google Cloud Storage to the local file system, assuming there is enough space to store it.

We’ll finish by cleaning up:

$ gsutil rm gs://baeldung-1-bucket/my-first-blob
Removing gs://baeldung-1-bucket/my-first-blob...
/ [1 objects]
Operation completed over 1 objects.
$ gsutil ls gs://baeldung-1-bucket/
$

rm (del works too) deletes the specified object

5. Conclusion

In this brief tutorial, we created credentials for Google Cloud Storage and connected to the infrastructure. We created a bucket, wrote data, and then read and modified it. As we’re working with the API, we also used gsutil to examine cloud storage as we created and read data.

We also discussed how to use buckets and write and modify data efficiently.