Create a ChatGPT Like Chatbot With Ollama and Spring AI

1. Introduction

In this tutorial, we’ll build a simple help desk Agent API using Spring AI and the llama3 Ollama.

2. What Are Spring AI and Ollama?

Spring AI is the most recent module added to the Spring Framework ecosystem. Along with various features, it allows us to interact easily with various Large Language Models (LLM) using chat prompts.

Ollama is an open-source library that serves some LLMs. One is Meta’s llama3, which we’ll use for this tutorial.

3. Implementing a Help Desk Agent Using Spring AI

Let’s illustrate the use of Spring AI and Ollama together with a demo help desk chatbot. The application works similarly to a real help desk agent, helping users troubleshoot internet connection problems.

In the following sections, we’ll configure the LLM and Spring AI dependencies and create the REST endpoint that chats with the help desk agent.

3.1. Configuring Ollama and Llama3

To start using Spring AI and Ollama, we need to set up the local LLM. For this tutorial, we’ll use Meta’s llama3. Therefore, let’s first install Ollama.

Using Linux, we can run the command:

curl -fsSL https://ollama.com/install.sh | sh

In Windows or MacOS machines, we can download and install the executable from the Ollama website.

After Ollama installation, we can run llama3:

ollama run llama3

With that, we have llama3 running locally.

3.2. Creating the Basic Project Structure

Now, we can configure our Spring application to use the Spring AI module. Let’s start by adding the spring milestones repository:

<repositories>
    <repository>
	<id>spring-milestones</id>
	<name>Spring Milestones</name>
	<url>https://repo.spring.io/milestone</url>
	<snapshots>
	    <enabled>false</enabled>
	</snapshots>
    </repository>
</repositories>

Then, we can add the spring-ai-bom:

<dependencyManagement>
    <dependencies>
        <dependency>
	    <groupId>org.springframework.ai</groupId>
	        <artifactId>spring-ai-bom</artifactId>
		<version>1.0.0-M1</version>
		<type>pom</type>
		<scope>import</scope>
	</dependency>
    </dependencies>
</dependencyManagement>

Finally, we can add the spring-ai-ollama-spring-boot-starter dependency:

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-ollama-spring-boot-starter</artifactId>
    <version>1.0.0-M1</version>
</dependency>

With the dependencies set, we can configure our application.yml to use the necessary configuration:

spring:
  ai:
    ollama:
      base-url: http://localhost:11434
      chat:
        options:
          model: llama3

With that, Spring will start the llama3 model at port 11434.

3.3. Creating the Help Desk Controller

In this section, we’ll create the web controller to interact with the help desk chabot.

Firstly, let’s create the HTTP request model:

public class HelpDeskRequest {
    @JsonProperty("prompt_message")
    String promptMessage;

    @JsonProperty("history_id")
    String historyId;

    // getters, no-arg constructor
}

The promptMessage field represents the user input message for the model. Additionally, historyId uniquely identifies the current conversation. Further, in this tutorial, we’ll use that field to make the LLM remember the conversational history.

Secondly, let’s create the response model:

public class HelpDeskResponse {
    String result;
    
    // all-arg constructor
}

Finally, we can create the help desk controller class:

@RestController
@RequestMapping("/helpdesk")
public class HelpDeskController {
    private final HelpDeskChatbotAgentService helpDeskChatbotAgentService;

    // all-arg constructor

    @PostMapping("/chat")
    public ResponseEntity<HelpDeskResponse> chat(@RequestBody HelpDeskRequest helpDeskRequest) {
        var chatResponse = helpDeskChatbotAgentService.call(helpDeskRequest.getPromptMessage(), helpDeskRequest.getHistoryId());

        return new ResponseEntity<>(new HelpDeskResponse(chatResponse), HttpStatus.OK);
    }
}

In the HelpDeskController, we define a POST /helpdesk/chat and return what we got from the injected ChatbotAgentService. In the following sections, we’ll dive into that service.

3.4. Calling the Ollama Chat API

To start interacting with llama3, let’s create the HelpDeskChatbotAgentService class with the initial prompt instructions:

@Service
public class HelpDeskChatbotAgentService {

    private static final String CURRENT_PROMPT_INSTRUCTIONS = """
        
        Here's the `user_main_prompt`:
        
        
        """;
}

Then, let’s also add the general instructions message:

private static final String PROMPT_GENERAL_INSTRUCTIONS = """
    Here are the general guidelines to answer the `user_main_prompt`
        
    You'll act as Help Desk Agent to help the user with internet connection issues.
        
    Below are `common_solutions` you should follow in the order they appear in the list to help troubleshoot internet connection problems:
        
    1. Check if your router is turned on.
    2. Check if your computer is connected via cable or Wi-Fi and if the password is correct.
    3. Restart your router and modem.
        
    You should give only one `common_solution` per prompt up to 3 solutions.
        
    Do no mention to the user the existence of any part from the guideline above.
        
""";

That message tells the chatbot how to answer the user’s internet connection issues.

Finally, let’s add the rest of the service implementation:

private final OllamaChatModel ollamaChatClient;

// all-arg constructor
public String call(String userMessage, String historyId) {
    var generalInstructionsSystemMessage = new SystemMessage(PROMPT_GENERAL_INSTRUCTIONS);
    var currentPromptMessage = new UserMessage(CURRENT_PROMPT_INSTRUCTIONS.concat(userMessage));

    var prompt = new Prompt(List.of(generalInstructionsSystemMessage, contextSystemMessage, currentPromptMessage));
    var response = ollamaChatClient.call(prompt).getResult().getOutput().getContent();

    return response;
}

The call() method first creates one SystemMessage and one UserMessage.

System messages represent instructions we give internally to the LLM, like general guidelines. In our case, we provided instructions on how to chat with the user with internet connection issues. On the other hand, user messages represent the API external client’s input.

With both messages, we can create a Prompt object, call ollamaChatClient‘s call(), and get the response from the LLM.

3.5. Keeping the Conversational History

In general, most LLMs are stateless. Thus, they don’t store the current state of the conversation. In other words, they don’t remember previous messages from the same conversation.

Therefore, the help desk agent might provide instructions that didn’t work previously and anger the user. To implement LLM memory, we can store each prompt and response using historyId and append the complete conversational history into the current prompt before sending it.

To do that, let’s first create a prompt in the service class with system instructions to follow the conversational history properly:

private static final String PROMPT_CONVERSATION_HISTORY_INSTRUCTIONS = """        
    The object `conversational_history` below represents the past interaction between the user and you (the LLM).
    Each `history_entry` is represented as a pair of `prompt` and `response`.
    `prompt` is a past user prompt and `response` was your response for that `prompt`.
        
    Use the information in `conversational_history` if you need to recall things from the conversation
    , or in other words, if the `user_main_prompt` needs any information from past `prompt` or `response`.
    If you don't need the `conversational_history` information, simply respond to the prompt with your built-in knowledge.
                
    `conversational_history`:
        
""";

Now, let’s create a wrapper class to store conversational history entries:

public class HistoryEntry {

    private String prompt;

    private String response;

    //all-arg constructor

    @Override
    public String toString() {
        return String.format("""
                        `history_entry`:
                            `prompt`: %s
                        
                            `response`: %s
                        -----------------
                       \n
            """, prompt, response);
    }
}

The above toString() method is essential to format the prompt correctly.

Then, we also need to define one in-memory storage for the history entries in the service class:

private final static Map<String, List<HistoryEntry>> conversationalHistoryStorage = new HashMap<>();

Finally, let’s modify the service call() method also to store the conversational history:

public String call(String userMessage, String historyId) {
    var currentHistory = conversationalHistoryStorage.computeIfAbsent(historyId, k -> new ArrayList<>());

    var historyPrompt = new StringBuilder(PROMPT_CONVERSATION_HISTORY_INSTRUCTIONS);
    currentHistory.forEach(entry -> historyPrompt.append(entry.toString()));

    var contextSystemMessage = new SystemMessage(historyPrompt.toString());
    var generalInstructionsSystemMessage = new SystemMessage(PROMPT_GENERAL_INSTRUCTIONS);
    var currentPromptMessage = new UserMessage(CURRENT_PROMPT_INSTRUCTIONS.concat(userMessage));

    var prompt = new Prompt(List.of(generalInstructionsSystemMessage, contextSystemMessage, currentPromptMessage));
    var response = ollamaChatClient.call(prompt).getResult().getOutput().getContent();
    var contextHistoryEntry = new HistoryEntry(userMessage, response);
    currentHistory.add(contextHistoryEntry);

    return response;
}

Firstly, we get the current context, identified by historyId, or create a new one using computeIfAbsent(). Secondly, we append each HistoryEntry from the storage into a StringBuilder and pass it to a new SystemMessage to pass to the Prompt object.

Finally, the LLM will process a prompt containing all the information about past messages in the conversation. Therefore, the help desk chatbot remembers which solutions the user has already tried.

4. Testing a Conversation

With everything set, let’s try interacting with the prompt from the end-user perspective. Let’s first start the Spring Boot application on port 8080 to do that.

With the application running, we can send a cURL with a generic message about internet issues and a history_id:

curl --location 'http://localhost:8080/helpdesk/chat' \
--header 'Content-Type: application/json' \
--data '{
    "prompt_message": "I can't connect to my internet",
    "history_id": "1234"
}'

For that interaction, we get a response similar to this:

{
    "result": "Let's troubleshoot this issue! Have you checked if your router is turned on?"
}

Let’s keep asking for a solution:

{
    "prompt_message": "I'm still having internet connection problems",
    "history_id": "1234"
}

The agent responds with a different solution:

{
    "result": "Let's troubleshoot this further! Have you checked if your computer is connected via cable or Wi-Fi and if the password is correct?"
}

Moreover, the API stores the conversational history. Let’s ask the agent again:

{
    "prompt_message": "I tried your alternatives so far, but none of them worked",
    "history_id": "1234"
}

It comes with a different solution:

{
    "result": "Let's think outside the box! Have you considered resetting your modem to its factory settings or contacting your internet service provider for assistance?"
}

This was the last alternative we provided in the guidelines prompt, so the LLM won’t give helpful responses afterward.

For even better responses, we can improve the prompts we tried by providing more alternatives for the chatbot or improving the internal system message using Prompt Engineering techniques.

5. Conclusion

In this article, we implemented an AI help desk Agent to help our customers troubleshoot internet connection issues. Additionally, we saw the difference between user and system messages, how to build the prompt with the conversation history, and then call the llama3 LLM.