Introduction to Triton Java API

1. Introduction

We use clever engineering to deploy complex machine learning models into production. While model research, training, and fine-tuning are predominantly performed in Python with frameworks such as PyTorch or TensorFlow, enterprise backends are often built in Java. Hence, we need a robust, scalable serving infrastructure to bridge this gap.

NVIDIA’s Triton Inference Server is an open-source model serving software that standardizes model deployment and execution. It provides a unified architecture capable of hosting models trained in almost any framework, such as ONNX, TensorFlow, PyTorch, and NVIDIA’s highly optimized TensorRT. Triton provides us with dynamic batching, concurrent model execution, and memory management while exposing a clean API for client applications.

In this tutorial, we’ll learn to interact with Triton Inference Server using Java to demonstrate object detection on images. We begin by reviewing the available APIs, then set up a local Triton instance using Docker, and finally write a concise Java client to execute a pre-trained YOLO model.

2. Overview of the Java API

We have two primary pathways for integrating Triton with Java applications.

2.1. The In-Process Java API

The In-Process Java API utilizes Java Native Interface (JNI) bindings to communicate directly with the underlying libtritonserver C library. Here, we bypass network protocols to embed the Triton server instance directly within the JVM process. We use this approach for low-latency edge deployments or network-starved environments.

Some of the important classes in this API are:

• TritonServer that represents the embedded server instance.
• TritonModel that represents a loaded model ready for inference.
• TritonRequest and TritonResponse that help us pass tensors back and forth in memory.

2.2. The Java Client API (gRPC/HTTP)

For most enterprise microservice architectures, we have models hosted on centralized CPU or GPU clusters. In such cases, we use the Java Client API via gRPC to access the models. Under the hood, Triton exposes a robust protobuf definition that Java applications can compile into strongly-typed stubs.

The key classes of gRPC API are:

• ManagedChannel that represents the underlying gRPC connection pool to the Triton server.
• InferenceServerBlockingStub that holds the synchronous client stub generated from Triton’s protobuf definitions.
• ModelInferRequest that encapsulates our input tensors, shapes, and data types.
• ModelInferResponse that gives the output containing the computed predictions.
• InferTensorContents to safely pack raw primitive types (like floats or integers) into byte streams.

2.3. Official Repository

For more advanced use, NVIDIA provides an official repository containing utility wrappers under client/src/java. It offers a cleaner TritonClient class that abstracts away the boilerplate protobuf generation and supports asynchronous inference and shared-memory execution.  Apart from this, some of the other advanced Triton features include:

• Asynchronous Inference: Using gRPC’s asynchronous stubs to handle high-throughput, non-blocking requests.
• Shared Memory: Allowing Triton and the Java client to read/write from the same system memory (or CUDA memory) space, eliminating the serialization and network overhead of gRPC.
• String Tensors: Passing text data to NLP models like BERT or LLaMA.

In this article, we’ll focus on building the raw gRPC Java Client API, as it’s the most common integration pattern for custom enterprise environments.

3. Local Setup With Docker

We’ll show standard object detection using the YOLO pretrained model running the Triton Inference Server. To run our example locally, we’ll use Docker Desktop to run an active instance of Triton Inference Server.

3.1. Prerequisites

Here are the prerequisites to run this setup on a local CPU-based machine:

• Docker Desktop.
• Java 11 or higher.
• Maven for dependency management.
• Python 3.10 and above.

3.2. Complete Project Structure

Here’s the complete directory structure for this project:

triton-java-yolo/
├── python/
│   ├── export_model.py              # It exports YOLO to ONNX 
│   └── requirements.txt             # Python dependencies (torch, ultralytics, onnx)
│
├── model_repository/                # It's a directory mounted into Docker Triton
│   └── yolo_onnx/
│       ├── 1/
│       │   └── model.onnx           # Compiled ONNX CPU runtime engine file
│       └── config.pbtxt             # Triton model configuration
│
├── src/
│   ├── main/
│   │   ├── proto/                   # Triton gRPC Protobuf definitions
│   │   │   ├── grpc_service.proto
│   │   │   └── model_config.proto
│   │   ├── java/
│   │   │   └── com/baeldung/triton/
│   │   │       ├── client/
│   │   │       │   └── TritonClientManager.java   
│   │   │       ├── yolo/
│   │   │       │   ├── ImagePreprocessor.java    
│   │   │       │   ├── YoloPostprocessor.java     
│   │   │       │   └── YoloInferenceRunner.java   
│   │   │       └── App.java                       
│   │   └── resources/
│   │       └── sample.jpeg           # Sample input image for object detection
│   └── test/
│       └── java/
│           └── com/baeldung/triton/
│               └── TritonInferenceLiveTest.java
│
├── pom.xml                          # Maven build file with gRPC & Protobuf plugins

3.3. Building the Model

First, we create the file export_model.py:

from ultralytics import YOLO
model = YOLO("yolov8n.pt")
model.export(format="onnx", imgsz=640, dynamic=False)
print("Export complete: yolov8n.onnx generated.")

Then, we build our Python virtual environment and use it to run the file export_model.py:

python export_model.py

This’ll download the pretrained model and then generate the ONNX runtime yolov8n.onnx in our working directory:

Downloading https://github.com/ultralytics/assets/releases/download/v8.4.0/yolovDownloading https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8n.pt to 'yolov8n.pt': 100% ━━━━━━━━━━━━ 6.2MB 42.1MB/s 0.1s
Ultralytics 8.4.34 🚀 Python-3.12.5 torch-2.2.2 CPU (Intel Core i9-9880H 2.30GHz)
YOLOv8n summary (fused): 72 layers, 3,151,904 parameters, 0 gradients, 8.7 GFLOPs

PyTorch: starting from 'yolov8n.pt' with input shape (1, 3, 640, 640) BCHW and output shape(s) (1, 84, 8400) (6.2 MB)

ONNX: starting export with onnx 1.22.0 opset 17...

3.4. Creating the Model Repository

Triton loads models from a strictly formatted directory structure known as the model repository. So, we’ll create the model_repository directory structure and  move and rename the ONNX file:

cp yolov8n.onnx ../model_repository/yolo_onnx/1/model.onnx

Thereafter, we’ll update the file config.pbtxt:

name: "yolo_onnx"
backend: "onnxruntime"
max_batch_size: 0

input [
  {
    name: "images"
    data_type: TYPE_FP32
    dims: [ 1, 3, 640, 640 ]
  }
]

output [
  {
    name: "output0"
    data_type: TYPE_FP32
    dims: [ 1, 84, 8400 ]
  }
]

 3.5. Run Server

With the repository prepared, let’s launch Triton Inference Server using the Docker container and mount our model_repository directory:

docker run --platform linux/amd64 --rm \ -p 8000:8000 -p 8001:8001 -p 8002:8002 \ -v /absolute/path/to/model_repository:/models \ nvcr.io/nvidia/tritonserver:23.10-py3 \ tritonserver --model-repository=/models

Here, we use Port 8000 for HTTP REST requests, port 8001 for the gRPC endpoint in our Java application, and port 8002 for Prometheus metrics.

We can verify the container by looking at the logs:

=============================
== Triton Inference Server ==
=============================

NVIDIA Release 23.10 (build 72127154)
Triton Server Version 2.39.0

Copyright (c) 2018-2023, NVIDIA CORPORATION & AFFILIATES.  All rights reserved.

I0822 07:34:40.100142 1 server.cc:619] 
+-------------+-----------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Backend     | Path                                                            | Config                                                                                                                                                        |
+-------------+-----------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+
| onnxruntime | /opt/tritonserver/backends/onnxruntime/libtriton_onnxruntime.so | {"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","default-max-batch-size":"4"}} |
+-------------+-----------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

I0822 07:34:40.100196 1 server.cc:662] 
+-----------+---------+--------+
| Model     | Version | Status |
+-----------+---------+--------+
| yolo_onnx | 1       | READY  |
+-----------+---------+--------+

3.6. Configure Protobufs

To run our Java client, we need to set up the gRPC proto and the Maven dependencies. First, we place the official Triton proto files (grpc_service.proto and model_config.proto from the Triton Server repo) inside src/main/proto/:

curl -L https://raw.githubusercontent.com/triton-inference-server/common/main/protobuf/grpc_service.proto -o src/main/proto/grpc_service.proto

curl -L https://raw.githubusercontent.com/triton-inference-server/common/main/protobuf/model_config.proto -o src/main/proto/model_config.proto

4. Java Client

Next, we build our Java application.

4.1. Maven Dependencies

First, we need to declare our Maven dependencies for this project. For gRPC, we’ll use the core libraries: grpc-netty-shaded, grpc-protobuf, and grpc-stub, alongside javax.annotation-api. To compile the .proto files into Java classes automatically, we choose the os-maven-plugin and the protobuf-maven-plugin.

Here is our dependencies:

<dependencies>
    <!-- gRPC & Protobuf Dependencies -->
    <dependency>
        <groupId>io.grpc</groupId>
        <artifactId>grpc-netty-shaded</artifactId>
        <version>${grpc.version}</version>
    </dependency>
    <dependency>
        <groupId>io.grpc</groupId>
        <artifactId>grpc-protobuf</artifactId>
        <version>${grpc.version}</version>
    </dependency>
    <dependency>
        <groupId>io.grpc</groupId>
        <artifactId>grpc-stub</artifactId>
        <version>${grpc.version}</version>
    </dependency>
    <dependency>
        <groupId>javax.annotation</groupId>
        <artifactId>javax.annotation-api</artifactId>
        <version>1.3.2</version>
    </dependency>
</dependencies>

Next is our list of plugins:

<plugins>
    <!-- Compiles the .proto files in src/main/proto into Java classes -->
    <plugin>
        <groupId>org.xolstice.maven.plugins</groupId>
        <artifactId>protobuf-maven-plugin</artifactId>
        <version>0.6.1</version>
        <configuration>
            <protocArtifact>com.google.protobuf:protoc:${protobuf.version}:exe:${os.detected.classifier}</protocArtifact>
            <pluginId>grpc-java</pluginId>
            <pluginArtifact>io.grpc:protoc-gen-grpc-java:${grpc.version}:exe:${os.detected.classifier}</pluginArtifact>
        </configuration>
        <executions>
            <execution>
                <goals>
                    <goal>compile</goal>
                    <goal>compile-custom</goal>
                </goals>
            </execution>
        </executions>
    </plugin>
</plugins>

4.2. Establishing the Connection

We initiate communication by creating a ManagedChannel and instantiating a blocking stub:

ManagedChannel channel = ManagedChannelBuilder.forAddress("localhost", 8001)
  .usePlaintext() 
  .build();
GRPCInferenceServiceBlockingStub blockingStub = GRPCInferenceServiceGrpc.newBlockingStub(channel);

4.3. Verifying Server Health

It’s a best practice to query the server’s health status to ensure the model has been loaded successfully, so that we are good to send heavy inference requests:

public boolean isServerLive() {
    try {
        ServerLiveRequest request = ServerLiveRequest.newBuilder().build();
        ServerLiveResponse response = blockingStub.serverLive(request);
        return response.getLive();
    } catch (Exception e) {
        return false;
    }
}

4.4. Preparing the Input Tensor in Java

Here is the preprocessing logic using a standard BufferedImage:

InputStream is = ImagePreprocessor.class.getClassLoader().getResourceAsStream(resourcePath);
BufferedImage originalImage = ImageIO.read(is);
BufferedImage resizedImage = new BufferedImage(targetWidth, targetHeight, BufferedImage.TYPE_INT_RGB);
Graphics2D g = resizedImage.createGraphics();
g.drawImage(originalImage.getScaledInstance(targetWidth, targetHeight, Image.SCALE_SMOOTH), 0, 0, null);
g.dispose();

Before sending an image to the YOLOv8 model, we preprocess it to match the exact input shape and format as per the model configuration. Standard computer vision preprocessing involves resizing, normalizing, and reordering the color channels. YOLOv8 expects an input tensor of shape [1, 3, 640, 640]. This corresponds to a batch size of 1, 3 color channels (RGB), and a resolution of 640×640. Furthermore, the data must be in planar format (NCHW), in which we store all red pixels first, followed sequentially by all green and blue pixels. Finally, we extract and normalize the pixels into a float array tensorData:

int totalPixels = targetWidth * targetHeight;
float[] tensorData = new float[3 * totalPixels];
int rOffset = 0, gOffset = totalPixels, bOffset = 2 * totalPixels;
for (int y = 0; y < targetHeight; y++) {
    for (int x = 0; x < targetWidth; x++) { 
        int rgb = resizedImage.getRGB(x, y); 
        int r = (rgb >> 16) & 0xFF;
        int gVal = (rgb >> 8) & 0xFF;
        int b = rgb & 0xFF;
        int index = y * targetWidth + x;
        tensorData[rOffset + index] = r / 255.0f;
        tensorData[gOffset + index] = gVal / 255.0f;
        tensorData[bOffset + index] = b / 255.0f;
    }
}

4.5. Executing the Inference

With our input tensor ready, we now construct our ModelInferRequest.

The protobuf definitions suggest that output data can be extracted using outputTensor.getContents().getFp32ContentsList(), Triton optimizes performance by leaving this list empty for output responses, avoiding the massive performance penalty of deserializing hundreds of thousands of floats. Instead, Triton packs the underlying C++ memory buffer into the raw_output_contents field as a ByteString. We extract these bytes and wrap them in a Java ByteBuffer using Little Endian byte order to read the floats manually:

ModelInferRequest request = ModelInferRequest.newBuilder()
  .setModelName("yolo_onnx")
  .setModelVersion("1")
  .addInputs(inputTensor)
  .build();

ModelInferResponse response = blockingStub.modelInfer(request);

ByteString rawData = response.getRawOutputContents(0);
ByteBuffer buffer = rawData.asReadOnlyByteBuffer().order(ByteOrder.LITTLE_ENDIAN);

List resultList = new ArrayList<>(buffer.capacity() / 4);
while (buffer.hasRemaining()) {
    resultList.add(buffer.getFloat());
}

4.6. Non-Maximal Suppression

YOLOv8 returns a massive matrix of shape [1, C, N] where C=84 (COCO dataset classes) and N>8000 (different anchor boxes across the image). So, it means that when an object is clearly visible, multiple overlapping anchor boxes will report a high-confidence detection for the same object.

To prevent our application from reporting multiple bounding boxes for a single object, we apply Non-Maximum Suppression (NMS). NMS isolates the highest-confidence bounding box for an object and suppresses (discards) any other boxes that overlap it significantly. We determine overlap using Intersection over Union (IoU). If a lower-confidence box overlaps the highest-confidence box by more than our threshold (e.g., 50%), it’s treated as a duplicate and removed:

if (current.classId == next.classId) {
    if (calculateIoU(current, next) > 0.5f) {
        suppressed[j] = true;
    }
}

4.7. Final Run

Now, we tie all our components together in the main application class. We begin by exporting our YOLO model. Then we preprocess our sample cat image. After that, we run the Inference over gRPC and finally post-process and print the detection:

mvn clean compile  
mvn exec:java -Dexec.mainClass="com.baeldung.triton.App"

It yields a clean, optimized output, accurately identifying objects in the image without duplicate bounding boxes:

Connecting to Triton Inference Server at localhost:8001
Triton Server is live and ready.
Building inference request for model: yolo_onnx
Sending inference request to Triton...
Received 705600 data points. Parsing bounding boxes...
Raw boxes found before NMS: 8
Detected [cat] (Confidence: 83.7%) at Box [xMin=2.0, yMin=53.2]
Final valid objects detected: 1

5. Testing

We need to start the Triton Inference Server Docker container and ensure it is running on localhost:8001 with our yolo_onnx model loaded to run the Live Inference test.

In our Live Inference test, we first preprocess our sample cat image. Then carry on the inference with the YOLO model, which natively outputs a matrix of [1, 84, 8400]. Moving further, we flatten it to 705,600 elements and then run the post-processor so the test logs print the NMS cat detection:

public void givenValidImage_whenRunningInference_thenReturnsDetections() throws Exception {
    float[] inputTensor = ImagePreprocessor.preprocessFromResources("sample.jpeg", 640, 640);

    YoloInferenceRunner runner = new YoloInferenceRunner(clientManager.getStub(), "yolo_onnx");
    List<Float> outputs = runner.runInference(inputTensor);

    assertFalse(outputs.isEmpty(), "Inference output should not be empty");
    
    assertEquals(705600, outputs.size(), 
      "Output tensor should contain exactly 705,600 float elements");

    YoloPostprocessor.parseAndPrint(outputs);
}

6. Conclusion

In this article, we’ve studied the Triton Inference Server and its integration with Java.

In a nutshell, we explored how Triton Inference Server provides a powerful bridge between the Python-dominated ML ecosystem and Java microservices. By using Docker, we set up a local testing environment, constructed the necessary protobuf-based gRPC requests, and successfully executed an object detection pass using an ONNX CPU runtime engine.

As ML models grow in size and complexity, utilizing dedicated inference servers like Triton ensures our Java applications remain responsive, scalable, and easy to maintain.

As always, the complete code examples are available over on GitHub.