1. Overview
In modern applications that integrate with Large Language Models (LLMs), when users submit similar or rephrased prompts, we end up making redundant calls to the LLM, leading to unnecessary costs and higher latency.
Semantic caching addresses this challenge by storing the user’s query along with the LLM’s response in a vector store. When a new query arrives, we first check the vector store for semantically similar, previously answered questions. If a close match is found, we return the cached response, bypassing the original LLM call entirely.
In this tutorial, we’ll build a semantic caching layer using Spring AI and Redis.
2. Setting up the Project
Before we start implementing our semantic cache layer, we’ll need to include the necessary dependencies and configure our application correctly.
2.1. Configuring an Embedding Model
First, we’ll configure an embedding model that’ll convert natural language text into numeric vectors. For our demonstration, we’ll use an embedding model from OpenAI.
Let’s start by adding the necessary dependency to our project’s pom.xml file:
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-starter-model-openai</artifactId>
<version>1.0.3</version>
</dependency>
Here, we import Spring AI’s OpenAI starter dependency, which we’ll use to interact with an embedding model.
Next, let’s configure our OpenAI API key and specify the embedding model in our application.yaml file:
spring:
ai:
openai:
api-key: ${OPENAI_API_KEY}
embedding:
options:
model: text-embedding-3-small
dimensions: 512
We use the ${} property placeholder to load the value of our API key from an environment variable.
Additionally, we specify text-embedding-3-small as our embedding model with 512 dimensions. Alternatively, we can use a different embedding model, as the specific AI model or provider is irrelevant for this demonstration.
On configuring these properties, Spring AI automatically creates a bean of type EmbeddingModel for us.
2.2. Configuring Redis as Vector Store
Next, we’ll need a vector store to save our query embeddings and their corresponding LLM responses. We’ll use Redis for this purpose, but again, we can choose a vector store based on our requirements.
First, let’s add the required dependency:
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-starter-vector-store-redis</artifactId>
<version>1.0.3</version>
</dependency>
The Redis vector store starter dependency enables us to establish a connection with Redis and interact with it as a vector store.
Now, let’s configure the connection URL to enable our application to connect to the provisioned Redis instance:
spring:
data:
redis:
url: ${REDIS_URL}
We’re again using a property placeholder to load the Redis connection URL from an environment variable. It’s important to note that the URL should follow the format of redis://username:password@hostname:port.
Next, we’ll need to configure a few custom properties for our semantic cache implementation. We’ll store these properties in our project’s application.yaml file and use @ConfigurationProperties to map the values to a record:
@ConfigurationProperties(prefix = "com.baeldung.semantic.cache")
record SemanticCacheProperties(
Double similarityThreshold,
String contentField,
String embeddingField,
String metadataField
) {}
Here, the similarityThreshold determines how semantically similar a new query must be (on a scale of 0 to 1) to a cached query for it to be considered a match.
The contentField specifies the field name where we’ll store the original natural language query inside our vector store. The embeddingField stores the vector representation of this natural language query, and the metadataField stores the corresponding LLM’s answer.
Now, let’s define the values of these properties in our application.yaml:
com:
baeldung:
semantic:
cache:
similarity-threshold: 0.8
content-field: question
embedding-field: embedding
metadata-field: answer
We set a similarity threshold of 0.8, ensuring only highly similar queries trigger cache hits. Next, the three field names we’ve chosen clearly indicate what data each field contains.
With our properties defined, let’s create the beans required to interact with our vector store:
@Configuration
@EnableConfigurationProperties(SemanticCacheProperties.class)
class LLMConfiguration {
@Bean
JedisPooled jedisPooled(RedisProperties redisProperties) {
return new JedisPooled(redisProperties.getUrl());
}
@Bean
RedisVectorStore vectorStore(
JedisPooled jedisPooled,
EmbeddingModel embeddingModel,
SemanticCacheProperties semanticCacheProperties
) {
return RedisVectorStore
.builder(jedisPooled, embeddingModel)
.contentFieldName(semanticCacheProperties.contentField())
.embeddingFieldName(semanticCacheProperties.embeddingField())
.metadataFields(
RedisVectorStore.MetadataField.text(semanticCacheProperties.metadataField()))
.build();
}
}
First, we create a JedisPooled bean that Spring AI uses to communicate with Redis. We pass the connection URL we’ve configured in our application.yaml file using the auto-configured RedisProperties bean.
Next, we define our RedisVectorStore bean, passing the JedisPooled bean and the auto-configured EmbeddingModel bean. Additionally, we use our semanticCacheProperties bean to define our custom field names. The RedisVectorStore bean is the core class that we’ll use in the upcoming section to interact with our vector store.
3. Implementing Semantic Caching
With our configuration in place, let’s build the service responsible for saving to and searching our semantic cache.
3.1. Saving LLM Response to Cache
Let’s first create a method to save LLM responses:
@Service
@EnableConfigurationProperties(SemanticCacheProperties.class)
class SemanticCachingService {
private final VectorStore vectorStore;
private final SemanticCacheProperties semanticCacheProperties;
// standard constructor
void save(String question, String answer) {
Document document = Document
.builder()
.text(question)
.metadata(semanticCacheProperties.metadataField(), answer)
.build();
vectorStore.add(List.of(document));
}
}
Here, in our SemanticCachingService class, we define a save() method that takes a natural language question and its corresponding answer as input.
Inside our method, we create a Document object with the question as the main text content and store the answer in the metadata.
Finally, we use the add() method of the autowired vectorStore bean to save the document. The bean automatically generates an embedding for the document’s text — i.e., the question — and stores it alongside the question and answer in the configured semantic cache.
Now, let’s implement the search functionality to retrieve cached responses:
Optional<String> search(String question) {
SearchRequest searchRequest = SearchRequest.builder()
.query(question)
.similarityThreshold(semanticCacheProperties.similarityThreshold())
.topK(1)
.build();
List<Document> results = vectorStore.similaritySearch(searchRequest);
if (results.isEmpty()) {
return Optional.empty();
}
Document result = results.getFirst();
return Optional
.ofNullable(result.getMetadata().get(semanticCacheProperties.metadataField()))
.map(String::valueOf);
}
Here, in our search() method, we first build a SearchRequest instance. We pass the question as the query, set the similarityThreshold from our properties, and pass 1 to the topK() method to retrieve only the single best match.
Then, we pass our searchRequest to the similaritySearch() method of the vectorStore bean. Again, the bean automatically generates an embedding for the input question behind the scenes and searches our semantic cache for the most similar entry that meets our threshold.
If no similar entry was found, we simply return an empty Optional.
Alternatively, if a match is found, we extract the first Document from the results, extract the answer from its metadata, and return it wrapped in an Optional.
4. Testing Our Implementation
Finally, let’s write a simple test to verify that our semantic cache implementation works correctly:
String question = "How many sick leaves can I take?";
String answer = "No leaves allowed! Get back to work!!";
semanticCachingService.save(question, answer);
String rephrasedQuestion = "How many days sick leave can I take?";
assertThat(semanticCachingService.search(rephrasedQuestion))
.isPresent()
.hasValue(answer);
String unrelatedQuestion = "Can I get a raise?";
assertThat(semanticCachingService.search(unrelatedQuestion))
.isEmpty();
First, we save an original question and answer pair to the vector store using our semanticCachingService.
Then, we search using a rephrased version of the original question. Despite the different wording, our service recognizes the similarity and returns the cached answer.
Finally, we verify that an unrelated question whose semantic meaning is entirely different results in a cache miss.
5. Conclusion
In this article, we’ve explored implementing semantic caching using Spring AI.
We configured an embedding model from OpenAI to convert text into vector representations and set up Redis as a vector store to store and search these embeddings. Then, we built and tested a caching service that saves LLM responses and retrieves them for semantically similar queries, reducing cost and latency.
For our demonstration, we’ve kept things simple. We can find a more advanced example that builds semantic caching on top of a Retrieval-Augmented Generation (RAG) chatbot here.
As always, all the code examples used in this article are available over on GitHub.
