Serialization with FlatBuffers in Java

1. Introduction

In this tutorial, we’ll explore FlatBuffers in Java and perform serialization and deserialization using it.

2. Serialization in Java

Serialization is the process of converting Java objects into a stream of bytes that can be transmitted over a network or persist in a file. Java provides an inbuilt object serialization mechanism through the java.io.Serializable interface and the java.io.ObjectOutputStream and java.io.ObjectInputStream classes.

However, owing to its several downsides, including a complicated approach to dealing with complex object graphs and dependent classes, several libraries are available for serialization and deserialization in Java.

Some of the widely used Java serialization libraries include Jackson and Gson. A newer standard for object serialization format is Protocol Buffers. Protocol Buffers is a language-agnostic binary serialization format developed by Google. They are used in high-performance environments and distributed systems where efficiency and interoperability are critical.

3. FlatBuffers

FlatBuffers is an efficient cross-platform serialization library developed by Google. It supports several languages, such as C, C++, Java, Kotlin, and Go. FlatBuffers were created for game development; therefore, performance and low memory overheads are default considerations in its design.

FlatBuffers and Protocol Buffers are created by Google and are very similar binary-based data formats. Both of these formats support efficient high-speed serialization and deserialization. The primary difference is that FlatBuffers doesn’t need additional data unpacking to an intermediate data structure before access.

3.1. Introduction to the FlatBuffers Library

A complete FlatBuffers implementation consists of the following components:

• A FlatBuffer schema file
• A flatc compiler
• Serialization and deserialization code

The FlatBuffer schema file serves as a template for the structure of the data model we’ll use. The syntax for the schema file follows a similar pattern to that of C-type or other interface description language (IDL) formats. We need to define the schema and the flatc compiler, then compile the schema file.

3.2. Tables and Schemas

A FlatBuffer is a binary buffer that contains nested objects (such as structs, tables, and vectors) organized using offsets.

This arrangement allows data to be traversed in place, similar to traditional pointer-based data structures. However, unlike many in-memory data structures, FlatBuffers strictly adhere to rules of alignment and endianness (always little), ensuring cross-platform compatibility. Moreover, for table objects, FlatBuffers offers both forward and backward compatibility.

Tables in FlatBuffers are the most basic data structures used to represent complex structures with named fields. Tables are similar to classes or structs in some languages and support fields of several types, such as int, short, string, struct, vectors, and even other tables.

3.3. The Flatc Compiler

The flatc compiler is a crucial tool provided by FlatBuffers that generates code in various programming languages, such as  C++ and Java, to help serialize and deserialize data according to the schema. This compiler inputs the schema definition and generates code in the desired programming language.

In upcoming sections, we’ll compile our schema files to generate code. However, we need to build and set up our compiler first to be able to use it.

We start by cloning the flatbuffers library into our system:

$ git clone https://github.com/google/flatbuffers.git

Once the flatbuffers directory is created, we use cmake to build the library into an executable. CMake (Cross-platform Make)  is an open-source, platform-independent build system designed to automate the process of building software projects:

$ cd flatbuffers
$ mkdir build
$ cd build
$ cmake ..

This completes the flatc compiler build process. We can verify the success of the installation by printing the version:

$ ./flatc --version
flatc version 23.5.26

The compiled files are now stored under the /flatbuffers/build path, and the flatc executable is also available in the same directory. We’ll use this file to build all schema files, and therefore, we can create a shortcut or alias to this path.

4. Working With FlatBuffers

In this section, we’ll explore the FlatBuffers library by implementing our use case. Let’s consider that we are developing a game across different terrains such as the sea, mountain, and plain land. Each terrain has its own set of unique properties.

The terrain information is necessary to load the game level and needs to be transmitted across the network to the players. Efficient serialization and deserialization are a must.

4.1. Schema Definition

The first thing we should start with is defining our terrain schema type. A terrain is a table in our flatbuffer. It can have many attributes, such as a name (Sea, Land, Mountain, Desert, etc.), color, and position (in the form of 3d vector coordinates). The terrain can have an effect applied as well. For example, there might be a sandstorm in a desert or a flood in the land. The effect can be a separate table within the original schema.

With this understanding, let’s write our schema as follows:

namespace MyGame.baeldung;
enum Color:byte { Brown = 0, Red = 1, Green = 2, Blue = 3, White = 4 }
struct Vec3 {
  x:float;
  y:float;
  z:float;
}
table Terrain {
  pos:Vec3; // Struct.
  name:string;
  color:Color = Blue;
  navigation: string;
  effects: [Effect]
}

table Effect {
  name:string;
  damage:short;
}

root_type Terrain;

We have an enum for identifying the color of the terrain, a struct for the coordinates, and two tables, the Terrain and Effect, with Terrain being the root type.

4.2. Schema Compilation

The flatc compiler is ready, and we use it to compile our schema file terrain.fbs:

$ cd <path to schema>
$ flatc --java terrain.fbs

We should note that the flatc path might vary from system to system depending on the installation location described in the previous section.

4.3. Creating Objects and Perform Serialization

The schema has already been compiled and is ready to go. We can start creating some terrains for our game using the schema. As part of this example walkthrough, we’ll create a desert terrain and a few effects for our terrain.

To use FlatBuffers in Java, we need to add a Maven dependency:

<dependency>
    <groupId>com.google.flatbuffers</groupId>
    <artifactId>flatbuffers-java</artifactId>
    <version>23.5.26</version>
</dependency>

We can now import the flatbuffers library along with the generated files from our schema:

import MyGame.terrains.*;
import com.google.flatbuffers.FlatBufferBuilder;

The files generated as part of the compilation process go under the same path defined in the schema’s namespace section (MyGame in our case).

An Effect class is available for us to use as a result of the compilation, which provides a createEffect() method. We’ll use that to create our desired effect. We’ll start by creating a builder object with an initial buffer size of 1024 bytes:

FlatBufferBuilder builder = new FlatBufferBuilder(INITIAL_BUFFER);

int sandstormOffset = builder.createString("sandstorm");
short damage = 3;
int sandStorm = MyGame.terrains.Effect.createEffect(builder, sandstormOffset, damage);

We can add more effects in the same way.

Next, we create our desert terrain. Let’s assign a color to the terrain, and give it a name and its navigation location:

byte color = MyGame.terrains.Color.YELLOW;
int terrainName = builder.createString("Desert");
int navigationName = builder.createString("south");

We add more terrain metadata and the effects using the auto-generated static methods of the Terrain class:

int effectOffset = MyGame.terrains.Terrain.createEffectsVector(builder, effects);

startTerrain(builder);
addName(builder, terrainName);
addColor(builder, color);
addNavigation(builder, navigationName);
addEffects(builder, effectOffset);
int desert = endTerrain(builder);
builder.finish(desert);

Let’s now serialize our terrain and its effects in our flatbuffer. We can store the buffer or transmit it over the network to clients:

ByteBuffer buf = builder.dataBuffer();

4.4. Deserialisation Using FlatBuffers

Let’s deserialize the flatbuffer object and access the terrain. We’ll start with a serialized array of bytes created from the buffer, and we’ll convert it into a ByteBuffer buffer:

ByteBuffer buf = java.nio.ByteBuffer.wrap(buffer);

This allows us to get an accessor to the root Terrain object from the buffer and access all its attributes:

Terrain myTerrain = Terrain.getRootAsTerrain(buf)
Assert.assertEquals(terrain.name(), "Desert");
Assert.assertEquals(terrain.navigation(), "south");

The compiler-generated code shows that each of the entity’s attributes comes with an associated accessor. We can access the associated effects as well:

Effect effect1 = terrain.effectsVector().get(0);
Effect effect2 = terrain.effectsVector().get(2);
Assert.assertEquals(effect1.name(), "Sandstorm");
Assert.assertEquals(effect2.name(), "Drought");

4.5. Mutating FlatBuffers

FlatBuffers are mostly read-only, owing to their static template structure. However, we might face a scenario where we need to change something in a flatbuffer before sending it to another piece of code. Let’s say we want to update the damage score of a sandstorm effect from the existing value of 3 to 10.

In such cases, in-place mutation of flatbuffers comes in handy.

Mutation of a flatbuffer is only possible if we build the schema with a –gen-mutable argument:

$ ./../flatc --gen-mutable --java terrain.fbs

This provides us with a mutate() method on all the accessors, which we can use to modify the value of a flatbuffer in place:

Assert.assertEquals(effect1.damage(), 3);
effect1.mutateDamage((short) 10);
Assert.assertEquals(effect1.damage(), 10);

5. JSON Conversion Using FlatBuffers

The flatc compiler provides techniques to convert binary files to JSON and vice-versa. Let’s say we have a JSON file for our terrain. We can use the compiler to create a binary file out of the JSON file using the following code:

flatc --binary <template file> <json file>

$ flatc --binary terrain.fbs sample_terrain.json

Conversely, we can convert a binary file to a full-fledged JSON file as well:

flatc --json --raw-binary <template file> -- <binary file>

$ flatc --json --raw-binary terrain.fbs -- sample_terrain.bin

6. Benefits of Using FlatBuffers

The usage of this cross-platform serialization library comes with a plethora of benefits:

• FlatBuffers organizes hierarchical data in a flat binary buffer, which we can directly access without the overhead of parsing or unpacking
• Incremental changes to our data structure are automatically and cleanly incorporated, making it easy to maintain backward compatibility with our evolving models
• They are also efficient in terms of memory utilization, as we only need the memory space occupied by the buffer to access your data
• They leave a tiny code footprint. The generated code is minimal, and we only need a single small header as a dependency, making integration a breeze
• FlatBuffers are strongly typed; hence, we can catch errors in compile time

7. Conclusion

In this article, we explored the FlatBuffers library and its capabilities to serialize and deserialize complex data. We took a hands-on approach to implementing code using the library and looked at the benefits and use cases of flatbuffers.