1. Overview
When we first ran the whole harness, its weakest surviving piece was the seat that judges the work: a one-paragraph reviewer that read the change and gave it a thumbs-up. In this lesson I deepen that seat into a real independent judge, the full Evaluator contract. It runs in a fresh context, sees the diff, the ticket, and the Implementor’s handoff note, re-runs what it can and judges what it can’t, returns a verdict of pass or revise with reasons, and never edits.
Alongside it, the ticket takes one more step: its completion checks split into the ones the Evaluator can re-run and the ones it has to assess. By the end, we can dispatch a fresh-context judge that catches what the writing context waved through and hands back a verdict we can act on.
The harness at this lesson’s refinement lives on the module1-lesson4 checkpoint branch.
2. Self-Evaluation Bias
Start with the simplest way to check a change: ask the context that just wrote it whether it’s done. It’s tempting, since the writer knows the work best. It’s also the one context we can’t trust to answer. Asked to judge its own output, a model tends to defend it, praising work it just produced even when that work is thin. The verdict it returns isn’t a verdict; it’s the writer marking its own homework.
We watched a version of this in our very first run. The Evaluator there was already a fresh context, a separate subagent, yet its whole contract was a single paragraph that amounted to a thumbs-up. Reading the change, it returned its verdict and, in the same breath, named the exact gap the vague ticket had left open:
Verdict: done.
[...]
Minor, non-blocking note: combining backlog=true&sprint=<id> returns empty by
construction (mutually exclusive predicates), which is a sensible interpretation
the ticket doesn't constrain.It saw the problem and still called the work done. That’s a rubber stamp, and its independence didn’t save it, because its contract never asked it to be skeptical.
None of this makes the Implementor’s own self-check worthless. That self-check is how the Implementor knows it’s finished so it can stop, and it’s what drives its internal loop of fixing until it considers the work done. We want the Implementor to do its best and to believe it’s there. What we don’t do is take that belief as the verdict. A separate seat has to decide whether the work is actually done, and the whole reason that seat exists is that it never wrote the code.
This is one real trace, not a law; a capable model asked the same question might answer honestly. The point isn’t that self-evaluation always fails, but that we can’t rely on it, and a judge we trust can’t be the writer.
3. What an Independent Evaluator Buys You
So what do we actually want from a judging seat? One thing: a verdict we can act on. A pass we trust enough to move on, or a rejection specific enough to fix. The naive thumbs-up gave us neither, because the seat that produced it had no stake in being right and no obligation to be skeptical.
The fix is easy to state wrong. It isn’t simply “use a fresh context.” Our first Evaluator already had a fresh context and still rubber-stamped. Independence is necessary, but on its own it buys nothing: a fresh window carrying a thumbs-up contract is just a stranger who also says yes.
What we want is a fresh context that re-runs what it can, judges what it can’t, returns a verdict with reasons, and never touches the code. Independence plus a real contract. That combination is what turns a second opinion into a verdict, and it seeds the idea this whole lesson turns on: a verdict from the context that wrote the code is not a verdict.
4. The Evaluator’s Contract
Here’s the seat rebuilt to that description. It matures the same one-paragraph evaluator.md we’ve carried since the naive harness; it doesn’t add a new role.
The contract has four moving parts. The Evaluator runs in a fresh context and is handed three things: the diff, the ticket, and the Implementor’s handoff note. It re-runs what it can and judges what it can’t. It returns a verdict, either pass or revise with reasons. And it never edits:
You are the Evaluator. You are given the diff, the ticket, and the Implementor's
handoff note. You did not write this change, and you do not take it on trust.
Re-run every completion check you can run yourself, the mechanical ones, and
confirm each result rather than believing the handoff note. Judge the checks you
cannot re-run, the acceptance criteria, against the diff and the ticket. A
behavioral change that ships without an automated test proving the new behavior
is not done, however green the existing suite is.
Return a verdict: pass, or revise. A pass means ship it. A revise must name
specific, actionable reasons a fresh pass can act on.
You never edit. You judge the work; you do not fix it. A revise goes back to a
fresh Implementor pass, and after three cycles without a pass, the run is the
human's problem.Each input earns its place, and none is trusted on faith. The diff shows what actually changed, not what the Implementor says changed. The ticket says what was asked and how “done” was defined. The handoff note carries the Implementor’s own account: what it did, what it verified, what it left uncertain. The Evaluator reads all three, then re-runs and re-judges rather than taking the note’s word.
One standing rule lives here and nowhere else. A behavioral change that ships without an automated test proving the new behavior is not done, however green the existing suite happens to be. This is an acceptance check the Evaluator enforces; it deliberately isn’t in the Implementor’s contract, so the Implementor can’t quietly satisfy it just by declaring itself finished.
The verdict shape is the visible upgrade over the thumbs-up. Revise isn’t a bare “needs work”; it carries named, actionable reasons, the specific things a next pass can fix. A reviewer that only says no is nearly as useless as one that only says yes.
The last clause, never edits, comes with a consequence the contract states in one line and the next sections unpack: the judge doesn’t fix what it rejects. Which raises the obvious question of who does.
5. Re-Running vs Judging: The Ticket’s Stage-3 Split
The contract keeps saying “re-runs what it can, judges what it can’t.” That line has a consequence for the ticket, and it’s why the ticket takes its next step now: the Evaluator is the first seat that actually needs to know which completion checks are which.
Sort them into two kinds. Mechanical checks are re-runnable facts: run them and get a yes or no. A suite is green. A test passes. A property binds. Acceptance criteria are judged calls the Evaluator can’t re-run: is this in scope, is the default sensible, does the change fit the code around it. Same conditions as before, now grouped by whether a machine settles them or a reader does.
Our configuration-default ticket, drafted as a flat list earlier, is the natural one to re-group. Its intent is unchanged; only the completion checks split, into two headings:
## Mechanical checks (the Evaluator re-runs)
- The existing suite passes unchanged: mvn test green, mvn checkstyle:check clean.
## Acceptance criteria (the Evaluator judges)
- A no-priority create yields the configured default, and an explicit priority
still overrides it.
- The default binds from the typed configuration surface (jiralite.tasks.default-priority).
- MEDIUM is a sensible default that preserves existing create behavior, and the
change stays in scope: the create path only, other fields untouched.
- The new setting is coherent with the existing jiralite.* configuration surface.The two kinds aren’t sealed off from each other, and the seam is the interesting part. Take the claim that a no-priority create yields the default. On its own that’s an acceptance criterion: someone reads the code and decides it’s true. Now bring in the Evaluator’s standing rule that a behavioral change must carry a test. Satisfy that rule and the same claim becomes a test the Evaluator can re-run for a yes or no. The test requirement is what manufactures a mechanical check: a judged behavioral question, once a test proves it, becomes a re-runnable fact. That rule lives in the Evaluator’s contract, not this ticket, which is exactly why the run in the next section can catch a change that skipped it.
We stop the split here. The mechanical checks stay a single kind; we don’t yet separate the durable tests that join the build from the one-time verifications we run and discard. That distinction is real, but it earns its place later, when a specific need calls for it.
6. The Revise Loop and the Three-Cycle Cap
Back to the question the contract left hanging. The Evaluator returns revise with reasons and, by its own contract, won’t touch the code. So a revise doesn’t get fixed where it’s raised. It loops back to a fresh Implementor pass, briefed as before by the ticket’s path plus the named reasons. A clean window fixes the work without defending the attempt that just fell short; the judge stays a judge.
That loop could run forever if a stubborn ticket kept failing. So we bound it with one plain rule: after three revise cycles without a pass, the loop stops and the ticket becomes the human’s problem. No gate, no state machine, just a sentence in the contract so the harness doesn’t spin on the same ticket indefinitely. Three is a number we can revisit; having a hard cap at all is the point.
Notice who’s running this loop: still us, by hand, exactly as since the naive harness. The seat that would automate that dispatching is a refinement we haven’t reached; today we’re deepening the Evaluator, and the Implementor stays as we last left it.
7. Running the Evaluator: Pass to Revise to Pass
Now we run the whole contract against a real ticket: the configuration-default change, a behavioral one. To show the loop on a real run rather than just describe it, we deliberately ran the Implementor on a weaker model while the Evaluator kept the capable default. Nothing was planted in the code; the diff and the handoff are the weak model’s, so whatever the Evaluator catches, it catches on its own. It took two Implementor cycles.
Cycle 1 — Implementor. The first pass came back fully green: 43 tests passing, checkstyle clean, both verified independently. The handoff was confident, and in its second half wrong: “bound via JiraLiteProperties.Tasks and injected as @Value.” Nothing read the typed record; the value flowed through the @Value field.
Cycle 1 — Evaluator. Judging from the ticket and diff rather than the note, the blind Evaluator returned revise with two reasons:
1. Typed configuration surface is decorative, not used. [...] the record
is dead code, so the typed-binding requirement is satisfied in
appearance only.
2. No test for the new behavior. [...] The green suite proves old behavior
is preserved, not that defaulting works.That’s the stage-3 split working live: the mechanical side came back green, the acceptance side caught the change.
Cycle 2 — Implementor. A fresh pass fixed both reasons: it deleted the @Value field, constructor-injected JiraLiteProperties and read properties.tasks().defaultPriority(), and added the positive-plus-negative test pair through the public HTTP surface.
Cycle 2 — Evaluator. Re-judging the new diff, it returned pass:
Verdict: pass
- Reason 1 (decorative typed config): the @Value field is gone [...] now
constructor-injects JiraLiteProperties and reads
properties.tasks().defaultPriority().
- Reason 2 (no test): adds taskCreatedWithoutPriorityUsesDefaultMedium and
explicitPriorityOverridesDefault, both through the public HTTP surface.It earned that pass rather than rubber-stamping the retry: it ran a scope check on a collaborator the fresh pass had added, confirming it was necessary and purely delegating. The added test re-runs green as a mechanical check:
+ void taskCreatedWithoutPriorityUsesDefaultMedium() throws Exception {
+ ...
+ .andExpect(jsonPath("$.priority").value("MEDIUM"));
+ }The capable Evaluator still caught what the under-powered Implementor missed, which hints at where to spend capability. The judging seat wants a high-capability, skeptical model, since it is the last line deciding whether the work is really done; the Implementor wants a solid mid-capability one. The basic model here only forced the gaps into the open; it is not the tier you would actually give the Implementor. Pinning a specific model to each seat is a later refinement, but this capability split is guidance that holds now.
The weak model’s second handoff still reported “what’s uncertain: None.” The self-check never got more honest; the independent verdict is what caught the gaps. A verdict from the context that wrote the code is not a verdict.
8. Conclusion
The judging seat is no longer a thumbs-up. It’s a fresh context that re-runs what it can, judges what it can’t, returns a verdict with named reasons, and never touches the code, backed by a ticket that says which checks are re-run and which are assessed. The difficulty didn’t vanish; it moved into a contract we write once and reuse on every run. The lesson holds in one line: a verdict from the context that wrote the code is not a verdict.
And the same seat keeps doing the dispatching. We hand the ticket to the Implementor, read the handoff, hand the diff to the Evaluator, read the verdict, and on a revise we go around again, all by hand. Every refinement so far has made the workers better while leaving that middle seat as manual as it was on day one. By now the repetition is impossible to miss, which is what tells us which seat is naive next