1. Overview
In this tutorial, we’ll learn to use Spring AI to integrate with Models offered by the LPU-based Groq AI inference engine, building a chatbot.
Groq exposes REST APIs that applications can invoke to consume its services. Additionally, SDKs are available in multiple programming languages, such as Python and JavaScript. In Python, popular libraries like LangChain and LiteLLM also support it. Furthermore, the Vercel AI SDK is a JavaScript-based library that has a module for integration with Groq.
Applications can comfortably switch from OpenAI to Groq AI services because Groq is fully compatible with OpenAI client libraries. As a result, Spring AI’s OpenAI chat client can connect to Groq with minimal configuration changes. Moreover, Spring AI doesn’t provide any separate library for Groq.
2. Prerequisites
Spring AI framework provides integration with a myriad of LLM services through its corresponding starter libraries. Similarly, for integrating with Groq services, Spring Boot applications must import the Spring AI OpenAI starter library:
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-openai-spring-boot-starter</artifactId>
<version>1.0.0-M6</version>
</dependency>
Typically, we use the Spring Initializr tool to import the necessary libraries optimally.
Finally, we must register on the Groq cloud and create an API key to use in the Spring AI configuration:
When users register on the Groq cloud, they receive a free subscription to get started with the Groq APIs. However, Groq applies model-specific rate limits on the API requests to ensure fair usage and availability.
3. Key Spring AI Components and Configurations
To effectively utilize Spring AI’s OpenAI library for accessing the Groq service APIs, it’s essential to understand its key classes, such as OpenAiChatModel and OpenAiChatOptions. The OpenAIChatModel class is the client class that takes the Prompt object to call the underlying OpenAI services.
However, for this article, we’ll demonstrate connecting to the Groq service. Additionally, the OpenAiChatOptions class helps specify the model name available on Groq, temperature, maxTokens, and other essential properties. We use it when we have to override the default properties of the chat client by passing it as an argument to the OpenAiChatModel#call() method. Finally, there are a few more general Spring AI classes, such as Prompt and ChatResponse, to help create the chat prompt and receive the response, respectively.
To autoconfigure the OpenAiChatModel, we can specify Spring AI’s OpenAI chat configurations under the spring.ai.openai.chat namespace. At a minimum, we must override the spring.ai.openai.chat.base-url configuration to point to the Groq API’s endpoint, which is api.groq.com/openai. Also, we must set the Groq API key corresponding to the spring.ai.openai.chat.api-key property. As a best practice, we recommend reading it from an environment property or a secure vault and then setting it to the configuration key.
We can also set the URL and the API keys in spring.ai.openai.base-url and spring.ai.openai.api-key configurations. However, the keys under the namespace spring.ai.openai.chat takes precedence.
In this section, we’ll see how the OpenAiAutoConfiguration class creates an OpenAiChatModel bean with the configuration from the application properties file.
First, let’s make a few important OpenAI configuration entries in the application-groq.properties file:
spring.application.name=spring-ai-groq-demo
spring.ai.openai.base-url=https://api.groq.com/openai
spring.ai.openai.api-key=gsk_XXXX
spring.ai.openai.chat.base-url=https://api.groq.com/openai
spring.ai.openai.chat.api-key=gsk_XXXX
spring.ai.openai.chat.options.temperature=0.7
spring.ai.openai.chat.options.model=llama-3.3-70b-versatile
As discussed earlier, in the property file, we’ve configured the Groq API endpoint, API keys, and a large language model. Interestingly, we’ve also specified the API endpoint and keys in the spring.ai.openai namespace. Because a few of the Spring AI beans depend on them, and the application would fail to start in their absence.
Next, let’s define a custom Spring GroqChatService class that’ll be responsible for calling the Groq service:
@Service
public class GroqChatService {
@Autowired
private OpenAiChatModel groqClient;
public String chat(String prompt) {
return groqClient.call(prompt);
}
public ChatOptions getChatOptions() {
return groqClient.getDefaultOptions();
}
}
The OpenAiAutoConfiguration bean instantiates the OpenAiChatModel class with properties from the configuration file, along with a few predefined default properties. In the service class, we’ve autowired the Spring OpenAiChatModel bean. The GroqChatservice#chat() method uses its call() method to invoke the Groq service. Additionally, the GroqChatService#getChatOptions() method returns the ChatOptions object containing the chat client’s configurations.
Finally, let’s see the Groq chat client in action with the help of a Junit test:
void whenCallOpenAIClient_thenReturnResponseFromGroq() {
String prompt = """
Context:
Support Ticket #98765:
Product: XYZ Wireless Mouse
Issue Description: The mouse connects intermittently to my laptop.
I've tried changing batteries and reinstalling drivers,
but the cursor still freezes randomly for a few seconds before resuming normal movement.
It affects productivity significantly.
Question:
Based on the support ticket, what is the primary technical issue
the user is experiencing with their 'XYZ Wireless Mouse'?;
""";
String response = groqChatService.chat(prompt);
logger.info("Response from Groq:{}", response);
assertThat(response.toLowerCase()).isNotNull()
.isNotEmpty()
.containsAnyOf("laptop", "mouse", "connect");
ChatOptions openAiChatOptions = groqChatService.getChatOptions();
String model = openAiChatOptions.getModel();
Double temperature = openAiChatOptions.getTemperature();
assertThat(openAiChatOptions).isInstanceOf(OpenAiChatOptions.class);
assertThat(model).isEqualTo("llama-3.3-70b-versatile");
assertThat(temperature).isEqualTo(Double.valueOf(0.7));
}
The Spring groqChatService bean is autowired in the class containing the test method. The method invokes groqChatService#chat() with a prompt consisting of a question and a pertaining context embedded in it. The context includes information about a support ticket raised on a computer mouse. Mostly, in a real-world application, a context is retrieved corresponding to a user query from a vector DB. Following this, the Groq service responds with an answer referring to the context:
The primary technical issue the user is experiencing with their
'XYZ Wireless Mouse' is intermittent connectivity, resulting in the
cursor freezing randomly for a few seconds before resuming normal movement.
Finally, towards the end, the method asserts that the chat options, such as the model and temperature, match the configuration values from the property file.
5. Customize Groq Client
So far, we’ve used the Spring AI configurations in the application properties file to help autoconfigure the chat client. However, in real-world applications, we’ll mostly end up customizing it to set properties such as model, temperature, etc.
First, let’s define such a custom OpenAiChatModel bean in a Spring configuration class:
@Configuration(proxyBeanMethods = false)
public class ChatAppConfiguration {
@Value("${groq.api-key}")
private String GROQ_API_KEY;
@Value("${groq.base-url}")
private String GROQ_API_URL;
@Bean
public OpenAiChatModel customGroqChatClient() {
OpenAiApi groqOpenAiApi = new OpenAiApi.Builder()
.apiKey(GROQ_API_KEY)
.baseUrl(GROQ_API_URL)
.build();
return OpenAiChatModel.builder()
.openAiApi(groqOpenAiApi)
.build();
}
}
The ChatAppConfiguration#customGroqChatClient() method in the class builds an OpenAiChatModel bean using the low-level OpenAiApi class. Here, we read the API key and the URL from a property file. Moreover, we can also modify the class to include complex logics and read them from downstream systems. When the Spring Boot application starts, the chat client object is available as a Spring bean with the name customGroqChatClient.
Next, let’s define a Spring Boot service class where we can autowire the custom OpenAIChatModel bean that we built in the Spring configuration class:
@Service
public class CustomGroqChatService {
@Autowired
private OpenAiChatModel customGroqChatClient;
public String chat(String prompt, String model, Double temperature) {
ChatOptions chatOptions = OpenAiChatOptions.builder()
.model(model)
.temperature(temperature)
.build();
return customGroqChatClient.call(new Prompt(prompt, chatOptions))
.getResult()
.getOutput()
.getText();
}
}
In the chat() method, we set the chat client’s configurations, such as model and temperature, in the ChatOptions object. Further, we pass the prompt and the ChatOptions object into the customGroqChatClient#call() method and finally extract the response text from the ChatResponse object.
Moving on, this time let’s see the custom Groq client in action in a Junit test:
void whenCustomGroqClientCalled_thenReturnResponse() {
String prompt = """
Context:
The Eiffel Tower is one of the most famous landmarks
in Paris, attracting millions of visitors each year.
Question:
In which city is the Eiffel Tower located?
""";
String response = customGroqChatService.chat(prompt, "llama-3.1-8b-instant", 0.8);
assertThat(response)
.isNotNull()
.isNotEmpty()
.contains("Paris");
logger.info("Response from custom Groq client: {}", response);
}
The test initiates a call to the chat() method of the autowired customGroqChatService bean, passing the prompt (comprising context and a related question), model, and temperature. The CustomGroqChatService#call() method then provides an answer. Subsequently, we validate that the response accurately addresses the question, In which city is the Eiffel Tower located?.
Finally, let’s take a look at the response from Groq:
The Eiffel Tower is located in Paris, France.
6. Conclusion
In this article, we highlight the integration of the Groq inference engine with Spring AI’s OpenAI library. Furthermore, the library allows for the use of Groq’s tooling feature to register and invoke external tools for action.
Unfortunately, Groq has limitations supporting multimodal models, and hence, Spring AI also lacks the feature. Inherently, Groq is not fully compatible with OpenAI. Therefore, we must be aware of these constraints when using its API.
As usual, the code used in this article is available over on GitHub.
