Introduction to Apache Pinot

Refactor Java code safely — and automatically — with OpenRewrite.

Refactoring big codebases by hand is slow, risky, and easy to put off. That’s where OpenRewrite comes in. The open-source framework for large-scale, automated code transformations helps teams modernize safely and consistently.

Each month, the creators and maintainers of OpenRewrite at Moderne run live, hands-on training sessions — one for newcomers and one for experienced users. You’ll see how recipes work, how to apply them across projects, and how to modernize code with confidence.

Join the next session, bring your questions, and learn how to automate the kind of work that usually eats your sprint time.

Distributed systems often come with complex challenges such as service-to-service communication, state management, asynchronous messaging, security, and more.

Dapr (Distributed Application Runtime) provides a set of APIs and building blocks to address these challenges, abstracting away infrastructure so we can focus on business logic.

In this tutorial, we'll focus on Dapr's pub/sub API for message brokering. Using its Spring Boot integration, we'll simplify the creation of a loosely coupled, portable, and easily testable pub/sub messaging system:

>> Flexible Pub/Sub Messaging With Spring Boot and Dapr

1. Introduction

Apache Pinot, developed originally by LinkedIn, is a real-time distributed OLAP (Online Analytical Processing) datastore designed to deliver low latency and high throughput for analytical queries. In this article, we’ll explore Apache Pinot’s key features and architecture, and learn how to interact with it.

2. What Is Apache Pinot?

Apache Pinot is a real-time distributed OLAP (Online Analytical Processing) data store designed to deliver low latency and high throughput for analytical queries. It is optimized for handling large-scale, time-series data and is commonly used for analytics on event streams, logs, and other types of real-time data. Below are some of its key features:

Real-time and Batch Data Ingestion: Pinot can ingest data in real-time from streams like Kafka and in batches from sources like Hadoop or S3.
Low Latency Queries: Pinot is designed to handle complex OLAP queries with sub-second response times.
Scalability: It can scale horizontally by adding more servers to handle increased load.
Pluggable Indexing: Supports various indexing techniques like inverted index, sorted index, range index, and more to optimize query performance.
Schema Flexibility: Allows for evolving schemas without downtime.
Support for SQL-like Query Language: Provides a SQL-like language for querying data, making it accessible to users familiar with SQL.

3. Architecture

Apache Pinot comprises several key components that work together to provide real-time distributed OLAP capabilities. These components include:

Cluster: It collects the software processes and hardware resources required to ingest, store, and process data. The processes include controller, zookeeper, server, broker, and minion. Pinot uses Apache Zookeeper as a distributed metadata store and Apache Helix for cluster management.
Controller: The controller manages the cluster and coordinates tasks such as segment creation, routing, and data management. It also handles configuration management and cluster metadata.
Broker: The broker component is responsible for query routing. It receives queries from clients and routes them to the appropriate servers that hold the relevant data segments. The broker then aggregates and returns results to the client.
Server: The server stores and manages data segments, processes queries, and returns results to the broker. It’s responsible for the real-time ingestion and indexing of data.
Minion: The minion component handles background tasks such as data compaction, segment management, and offline segment generation. It offloads these tasks from the servers to ensure efficient resource utilization.
Tenant: A tenant enables multi-tenancy, allowing users or applications to share the cluster while maintaining data and resource isolation, ensuring fair usage and performance isolation.
Segment: Pinot stores data in segments, immutable files containing a subset of the dataset. Each segment optimizes for fast reads, utilizing techniques like columnar storage to improve query performance. Pinot replicates segments across multiple nodes to ensure data availability and fault tolerance.

When we submit a query, broker nodes distribute it to the appropriate server nodes containing the relevant data segments. The server nodes process the query and return results to the broker nodes, which aggregate the results and send them back to the client. This distributed query processing ensures efficient and quick execution of queries.

4. Installing Pinot

We can install Pinot using Docker, Kubernetes, or directly on our local machine. The official documentation provides detailed instructions for various installation methods. We’ll follow the installation using Docker.

To install Pinot via Docker, the system needs to pass the following criteria:

Docker must be installed on the machine.
The docker memory must be configured with a minimum of 4 CPUs, 16GM Memory, 4GB Swap, and 60 GB disk image size.

After setting up and running Docker, execute the following command in a terminal to fetch the latest image:

docker pull apachepinot/pinot:latest

5. Working with Pinot

Now that we’ve downloaded the docker image, let’s set up the cluster. Pinot offers quick start commands to launch instances of its components in a single process and import pre-built datasets.

Let’s take one of the examples from QuickStart which starts all the components and creates a table called baseballStats.

It initiates a standalone data ingestion job to build a segment from a specified CSV data file for the baseballStats table and uploads the segment to the Pinot Controller:

docker run \
    -p 2123:2123 \
    -p 9000:9000 \
    -p 8000:8000 \
    -p 7050:7050 \
    -p 6000:6000 \
    apachepinot/pinot:1.1.0 QuickStart \
    -type batch

In the above command, port 2123 is the Zookeeper port, 9000 is the Pinot Controller port, 8000 maps to the Broker port, 7050 is the server port, and 6000 is the Minion port.

We can manually set up a cluster by following the steps mentioned here. To verify if the setup is correct, access the Pinot controller at http://localhost:9000.

The image below provides an overview of the cluster, showing its overall health and status, along with details about the connected instances, including controllers, brokers, servers, and minions:

We can interact with the created table using the Query Console. This interface lists all available tables and includes a query editor for writing and executing queries. The same window displays the query results upon execution as shown below:

6. Conclusion

In this tutorial, we covered the basics of Apache Pinot and explored its architecture. Apache Pinot is a leading datastore for real-time analytics, enabling organizations to process and analyze large data volumes instantly. Its scalable architecture, low-latency queries, and versatility make it a top choice for businesses.

As demand for real-time insights grows, Apache Pinot drives innovation and transformation in the digital landscape.