Spring AI With Docker Model Runner

1. Introduction

Docker Model Runner, introduced in Docker Desktop 4.40 for Mac with Apple Silicon (at the time of writing this article), revolutionizes local AI development by simplifying the deployment and management of large language models (LLMs). It tackles common challenges such as complex setup processes, high cloud inference costs, and data privacy concerns.

By providing an OpenAI-compatible Inference API, Model Runner enables seamless integration with frameworks like Spring AI, allowing developers to build AI-powered applications locally with ease. In this tutorial, we’ll learn how to set up Docker Model Runner and create a Spring AI application that connects to it. By the end, we’ll have a fully functional local AI application leveraging a powerful LLM.

2. Docker Model Runner

Docker Model Runner is a tool designed to simplify the deployment and execution of LLMs inside Docker containers. It’s an AI Inference Engine offering a wide range of models from various providers.

Let’s see the key features that Docker Model Runner includes:

• Simplified Model Deployment: Models are distributed as standard Open Container Initiative (OCI) artifacts on Docker Hub under the ai namespace. This makes it easy to pull, run, and manage AI models directly within Docker Desktop.

• Broad Model Support: Supports a variety of LLMs from multiple providers, such as Mistral, LLaMA, and Phi-4, ensuring flexibility in model selection.

• Local Inference: Runs models locally, enhancing data privacy and eliminating dependency on cloud-based inference.

• OpenAI-Compatible API: Provides a standardized API that integrates effortlessly with existing AI frameworks, reducing development overhead.

3. Set Up Environment

This section outlines the prerequisites for using Docker Model Runner and Maven dependencies to create a Spring AI application that uses the Model Runner.

3.1. Prerequisites

To use Docker Model Runner, we’ll need a few things:

• Docker Desktop 4.40 or later: Installed on a Mac with Apple Silicon.

• Java 21 or later: Required for Spring AI development.

• Model Runner-compatible LLM: A model compatible with Docker Model Runner, such as LLaMA or Gemma 3.

3.2. Maven Dependencies

Let’s start by importing the spring-boot-starter-web, spring-ai-openai-spring-boot-starter, spring-ai-spring-boot-testcontainers, and junit-jupiter dependencies to the pom.xml:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
    <version>1.0.0-M6</version>
</dependency>
<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-openai-spring-boot-starter</artifactId>
    <version>1.0.0-M6</version>
</dependency>
<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-spring-boot-testcontainers</artifactId>
    <version>1.0.0-M6</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>junit-jupiter</artifactId>
    <version>1.19.8</version>
    <scope>test</scope>
</dependency>

4. Enabling and Configuring Docker Model Runner

This section outlines the steps to enable Docker Model Runner and pull a specific model using two distinct methods.

4.1. Enable Model Runner With a Specific TCP Port

First, let’s  enable Model Runner and expose it on a specific TCP port (e.g., 12434):

docker desktop enable model-runner --tcp 12434

This configures Model Runner to listen on http://localhost:12434/engines. In our Spring AI application, we need to configure the api-key, model, and base URL to point to the Model Runner endpoint:

spring.ai.openai.api-key=${OPENAI_API_KEY}
spring.ai.openai.base-url=http://localhost:12434/engines
spring.ai.openai.chat.options.model=ai/gemma3

4.2. Enable Model Runner With Testcontainers

We can run the following command to enable Model Runner without specifying a port:

docker desktop enable model-runner

This sets up Model Runner to run on the default internal Docker network. Then, we use Testcontainers and set the base-url, api-key, and model as follows:

@TestConfiguration(proxyBeanMethods = false)
class TestcontainersConfiguration {
    @Bean
    DockerModelRunnerContainer socat() {
        return new DockerModelRunnerContainer("alpine/socat:1.8.0.1");
    }

    @Bean
    DynamicPropertyRegistrar properties(DockerModelRunnerContainer dmr) {
        return (registrar) -> {
          registrar.add("spring.ai.openai.base-url", dmr::getOpenAIEndpoint);
          registrar.add("spring.ai.openai.api-key", () -> "test-api-key");
          registrar.add("spring.ai.openai.chat.options.model", () -> "ai/gemma3");
        };
    }
}

The provided TestcontainersConfiguration class is a Spring Boot @TestConfiguration designed for integration testing with Testcontainers. It defines two beans: a DockerModelRunnerContainer that starts a Docker container using the alpine/socat:1.8.0.1 image, likely to proxy or mock an AI service endpoint, and a DynamicPropertyRegistrar that dynamically sets Spring AI properties. These properties configure the AI client with a base URL from the container’s endpoint (via getOpenAIEndpoint()), a test API key (test-api-key), and a model identifier (ai/gemma3). The @TestConfiguration(proxyBeanMethods = false) annotation ensures lightweight bean creation for testing without proxying. This setup enables tests to simulate an AI service environment without external dependencies, using the socat container to handle requests. The socat forwards traffic to the internal model-runner.docker.internal service.

4.3. Pulling and Verifying the Gemma 3 Model

Now, after enabling Model Runner using one of the options, we pull the Gemma 3 model:

docker model pull ai/gemma3

Then, we can confirm it’s available locally:

docker model list

This command lists all locally available models, including ai/gemma3.

5. Integration With Spring AI

Now, let’s create a simple controller to interact with the model:

@RestController
class ModelRunnerController {
    private final ChatClient chatClient;

    public ModelRunnerController(ChatClient.Builder chatClientBuilder) {
        this.chatClient = chatClientBuilder.build();
    }

    @GetMapping("/chat")
    public String chat(@RequestParam("message") String message) {
        return this.chatClient.prompt()
          .user(message)
          .call()
          .content();
    }
}

5.1. Testing Model Runner With a Specific TCP Port

To use Docker Model Runner, we need to configure the OpenAI client to point to the right endpoint and use the model pulled earlier. Now, we start the application and test the /chat endpoint:

curl "http://localhost:8080/chat?prompt=What%20is%20the%20future%20of%20AI%20development?"

The response will be generated by the Gemma 3 model running in Model Runner.

5.2. Testing Model Runner With Testcontainers

Let’s create the ModelRunnerApplicationTest class. It will import the TestcontainersConfiguration class and call the sample controller:

@Import(TestcontainersConfiguration.class)
class ModelRunnerApplicationTest {
    // ...

    @Test
    void givenMessage_whenCallChatController_thenSuccess() {
        // given
        String userMessage = "Hello, how are you?";

        // when
        ResponseEntity<String> response = restTemplate.getForEntity(
          baseUrl + "/chat?message=" + userMessage, String.class);

        // then
        assertThat(response.getStatusCode().is2xxSuccessful()).isTrue();
        assertThat(response.getBody()).isNotEmpty();
    }
}

The @Import(TestcontainersConfiguration.class) imports the TestcontainersConfiguration class, which defines a DockerModelRunnerContainer (running alpine/socat:1.8.0.1). Also, it dynamically registers Spring AI properties (e.g., spring.ai.openai.base-url, spring.ai.openai.api-key, spring.ai.openai.chat.options.model). This ensures the test environment is configured with a mock AI service endpoint provided by the Testcontainers-managed container.

6. Conclusion

Docker Model Runner provides a developer-friendly, privacy-focused, and cost-effective solution for running LLMs locally, particularly for those building GenAI applications within the Docker ecosystem. In this article, we explored Docker Model Runner’s capabilities and demonstrated its integration with Spring AI.