Building LLM-as-a-Judge Using Recursive Advisors in Spring AI

1. Overview

With Spring AI, we can integrate LLMs into our Spring applications. This is helpful, for example, for processing unstructured data like wiki pages, emails, or chat messages by extracting structured information to store it in a database. However, LLM responses are not deterministic and so neither testable nor handable by our application code. The best way to implement a kind of quality gate for LLM responses is to use an LLM again for evaluation. This pattern is called LLM-as-a-judge.

In this tutorial, we’ll learn how to implement the LLM-as-a-Judge pattern in Spring AI.

2. Maven Dependencies

We add the Spring AI OpenAI starter to our pom.xml:

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-openai</artifactId>
    <version>${spring-ai.version}</version>
</dependency>

The latest version of spring-ai-starter-model-openai is available on Maven Central.

3. The LLM-as-a-Judge Pattern

The LLM-as-a-Judge pattern is like a code review. A developer (the generator) writes code, and a senior engineer (the judge) reviews it and gives structured feedback. If the judge decides that the code doesn’t meet the requirements, the developer needs to improve it, and the judge then needs to review it again.

Applied to LLMs, the process is:

• The generator produces an initial answer to the user’s question
• The judge evaluates that answer against a rubric and returns a numeric score and written feedback
• If the score is too low, the generator receives the original question plus the judge’s critique and produces a refined answer.
• This will repeat until the answer meets the requirements. To avoid endless repetition in case of never reaching the needed quality, it’s a best practice to limit the number of loops.

This pattern improves output quality without model fine-tuning, without a human in the loop, and without changing a single line of calling code.

It’s most valuable wherever a weak answer has real consequences, e.g., customer support bots, where a vague answer to a billing question damages trust directly. In such cases, the pattern acts as an invisible quality gate between the model and the user.

3.1. Why Recursive Advisors Are the Right Fit

Advisors in Spring AI are interceptors that wrap the request/response cycle of a ChatClient. A CallAroundAdvisor can inspect a request before it reaches the model and modify the response before it reaches the caller, but it only ever makes one model call per invocation.

Recursive advisors go further: they hold an internal ChatClient reference and can make additional model calls from within the advisor itself.

We can apply that capability to a specific problem: using the second model call as a judge, not just an extension. The generate-judge-refine loop becomes a single, self-contained component that’s completely transparent to the caller.

The caller just invokes chatClient.prompt().user(“…”).call().content() as usual. The judge’s logic is invisible.

4. Implementing the LLM-as-a-Judge Advisor

Let’s build the feature in three parts: a value object for the verdict, a prompt template for the judge, and the advisor itself.

4.1. The Verdict Record

The judge returns a structured verdict. We model it as a simple Java record:

public record Verdict(double score, String feedback) {}

In our sample, the score is a value between 0.0 (poor) and 1.0 (excellent). The feedback explains what’s missing or weak, and it’s what we’ll feed back to the generator during refinement.

4.2. The Judge Prompt Template

The judge needs a clear, structured system prompt. For this, let’s define it as a constant inside the advisor:

private static final String JUDGE_SYSTEM_PROMPT = """
    You are a strict quality evaluator for AI-generated answers.
    
    Given a user question and an AI-generated answer, rate the answer quality.
    
    Use this rubric:
    - 1.0: Complete, accurate, and clearly explained
    - 0.7: Mostly correct but missing details or clarity
    - 0.4: Partially correct or overly vague
    - 0.0: Incorrect, irrelevant, or harmful
    
    Respond ONLY with a valid JSON object. Do not add any explanation outside the JSON.
    Format: {"score": <0.0 to 1.0>, "feedback": "<one concise sentence>"}
    """;

It’s important to declare an explicit rubric and a strict JSON output to get a parseable and consistent answer from the judge model.

4.3. The Advisor Class and Its Dependencies

The key design decision of a recursive advisor is that it holds its own ChatClient instance to make additional model calls independently of the advisor chain. We also inject the configurable quality thresholds:

public class LlmJudgeAdvisor implements CallAdvisor {

    private final ChatClient judgeClient;
    private final double scoreThreshold;
    private final int maxRefinements;

    public LlmJudgeAdvisor(
      ChatClient judgeClient,
      double scoreThreshold,
      int maxRefinements
    ) {
        this.judgeClient = judgeClient;
        this.scoreThreshold = scoreThreshold;
        this.maxRefinements = maxRefinements;
    }

    @Override
    public int getOrder() {
        return 0;
    }

    @Override
    public String getName() {
        return "LlmJudgeAdvisor";
    }

    // ...
}

The ChatClient.Builder receives the same auto-configured model settings as the rest of the application. We could wire a different, specialized judge model here. The official Spring AI docs recommend this specifically to avoid the model judging its own output too leniently.

4.4. Evaluating and Refining the Response

The adviseCall() method is the only entry point that the advisor chain calls. Rather than forwarding the request once, we call chain.copy(this).nextCall(request), which creates a sub-chain that includes our advisor again. That’s what enables the loop to re-evaluate and re-generate across multiple attempts:

@Override
public ChatClientResponse adviseCall(ChatClientRequest request, CallAdvisorChain chain) {
    for (int attempt = 1; attempt <= maxRefinements + 1; attempt++) { 
        ChatClientResponse response = chain.copy(this).nextCall(request);
        if (attempt > maxRefinements) {
          return response;
        }
        Verdict verdict = evaluate(request, response); 
        if (verdict.score() >= scoreThreshold) {
            return response;
        }

        request = addFeedback(request, verdict.feedback());
    }
    return chain.copy(this).nextCall(request);
}

The loop needs an explicit upper bound to avoid unbounded recursion. On the last allowed attempt, the method returns the response immediately, skipping the evaluation. There’s no point scoring an answer we can’t act on. Otherwise, if the score meets scoreThreshold, it returns early. If not, it augments the request with the judge’s feedback and continues to the next iteration.

4.5. Evaluating the Answer

The evaluate() method sends the original question and the generated answer to the judge model. We can use  .entity(…) to render the response JSON directly into our record, without any manual JSON parsing:

private Verdict evaluate(ChatClientRequest request, ChatClientResponse response) {
    String question = request.prompt().getUserMessage().getText();
    String answer = response.chatResponse().getResult().getOutput().getText();

    return judgeClient.prompt()
      .system(JUDGE_SYSTEM_PROMPT)
      .user("Question: " + question + "\n\nAnswer: " + answer)
      .call()
      .entity(Verdict.class);
}

Spring AI injects a JSON schema derived from the Verdict record into the model call and handles deserialization. This removes the need for any try-catch around JSON parsing and keeps the evaluation code focused on the prompt, not on format handling.

4.6. Augmenting the Request With Feedback

When the rating falls short, we don’t start over with the original prompt. Instead, let’s augment it, i.e., we append the judge’s critique to the user message so the model has full context for its next attempt. The addFeedback() helper does this using ChatClientRequest.mutate():

private ChatClientRequest addFeedback(ChatClientRequest original, String feedback) {
    Prompt augmented = original.prompt()
      .augmentUserMessage(msg -> msg.mutate()
          .text(msg.getText()
              + "\n\nYour previous answer was insufficient. Feedback: " + feedback
              + "\nPlease provide an improved answer.")
          .build());
    return original.mutate().prompt(augmented).build();
}

augmentUserMessage() gives us a mutable copy of the user message without touching the rest of the request. System prompt, tool definitions, and conversation history all stay intact. The updated ChatClientRequest is passed into the next loop iteration.

5. Configuring the Application

Let’s declare both beans in a single @Configuration class. The LlmJudgeAdvisor bean receives its own ChatClient instance, built from the same auto-configured ChatClient.Builder, along with the quality thresholds from application.properties:

@Configuration
public class ChatConfig {

    @Bean
    public ChatClient chatClient(
      ChatClient.Builder builder, 
      LlmJudgeAdvisor judgeAdvisor
    ) {
        return builder
          .defaultAdvisors(judgeAdvisor)
          .build();
    }

    @Bean
    public LlmJudgeAdvisor llmJudgeAdvisor(
      ChatClient.Builder builder,
      @Value("${judge.score-threshold:0.7}") double scoreThreshold,
      @Value("${judge.max-refinements:2}") int maxRefinements
    ) {
        return new LlmJudgeAdvisor(
          builder.build(),
          scoreThreshold,
          maxRefinements
        );
    }

}

We can then set the thresholds in application.properties:

spring.ai.openai.api-key=${OPENAI_API_KEY}
spring.ai.openai.chat.options.model=gpt-4o-mini

judge.score-threshold=0.75
judge.max-refinements=2

A score threshold of 0.75 means the judge must rate the answer at least 75% quality before we accept it. Setting the number of maximum refinements to 2 caps the recursive loop at two improvement attempts, preventing runaway API usage.

6. Testing the Advisor

Let’s implement a @SpringBootTest with a single mocked ChatModel. Both the generator and the judge use the same underlying instance, so one mock covers the entire call sequence. No API key is needed.

The test stubs four sequential chatModel responses matching the expected flow:

@SpringBootTest
class LlmJudgeAdvisorTest {

    @MockitoBean
    ChatModel chatModel;

    @Autowired
    ChatClient chatClient;

    @Test
    void givenLowQualityAnswer_whenAdvisorRuns_thenAnswerIsRefined() {
        when(chatModel.call(any(Prompt.class)))
          .thenReturn(buildChatResponse("It runs Java."))
          .thenReturn(buildChatResponse("{\"score\": 0.3, \"feedback\": \"Too vague.\"}"))
          .thenReturn(buildChatResponse(
            "The JVM executes Java bytecode, manages memory, and enables platform independence."))
          .thenReturn(buildChatResponse("{\"score\": 0.9, \"feedback\": \"Complete and accurate.\"}"));

        String result = chatClient.prompt()
          .user("Explain what a JVM is.")
          .call()
          .content();

        assertThat(result).contains("bytecode");
    }

    private ChatResponse buildChatResponse(String content) {
        return new ChatResponse(List.of(new Generation(new AssistantMessage(content))));
    }
}

The four stubs map directly to the generate-evaluate-refine-evaluate cycle: call one is the weak generator response, call two is the judge’s low verdict, call three is the refined generator response, and call four is the judge’s passing verdict. The chatClient with the advisor is real. Only the model underneath is mocked. This means the full advisor logic runs as it would in production.

7. Conclusion

In this article, we implemented the LLM-as-a-Judge pattern in Spring AI using a recursive CallAdvisor. We saw how the generate-evaluate-refine loop maps naturally onto the advisor model: the first model call produces an answer, Spring AI’s structured output maps the judge’s response into a typed Verdict, and the loop augments the original request with feedback until the score is sufficient or the attempt limit is reached.

The result is a reusable component that improves output quality automatically and remains completely transparent to any caller using the ChatClient.

The code for this tutorial is available over on GitHub.