Skip to content
Sign in
Artwork for The Sam Ellis Show
The Sam Ellis Show · Wednesday · 9 min

The Test Suite Became a Feedback Loop

A test can score an agent’s last run. A feedback loop has to make the next run better. Sam interviews Mark Pollack, founder of Spring AI, about Agent Control Theory: record the work, judge it against explicit requirements, measure the result and feed the evidence into the next attempt. Pollack’s examples show why each part matters. A code-coverage prompt changed where an agent spent its effort without improving quality or cost. All 290 tests passed in a commonly used Spring PetClinic sample application. JetBrains developer Anton Arhipov wrote the feature specification, and his review missed a required lock order that an AI jury flagged and Pollack later confirmed. That established a lock-order violation, not a reproduced deadlock. A separate Agent Client Protocol SDK evaluation moved one cancellation requirement from failure to a passing rerun after an identifiable fix. The loop does not eliminate judgment. Pollack says deterministic checks should come first, model judges should be grounded in evidence, and a human still has to catch a judge that is confidently measuring the wrong thing. Sources Pollack AI Spring Framework overview Spring PetClinic sample application Mark Pollack — fixed-commit PetClinic case study Mark Pollack — PetClinic case-study write-up Agent Client Protocol introduction Mark Pollack — Agent Control Theory talk Andrew Ng — Agentic AI course Pollack AI Lab — Code Coverage v3: The Exemplar Effect Agent Client Protocol pending-permission requirement Java SDK failing evaluation Java SDK cancellation fix Java SDK passing rerun DevNexus biography for Mark Pollack Email correspondence with the show: Mark Pollack, AgentWorks and Agent Judge If you supervise coding agents, what evidence changed the next run rather than merely scoring the last one? Email SamEllisShow@protonmail.com. Suggested subject line: “The feedback loop caught this.” Anonymous notes and source-protection requests are welcome. Send a source tip: SamEllisShow@protonmail.com. How this episode was checked: an AI fact-checker and an AI editorial reviewer checked the script, and a human editor approved the exact script and the finished audio before publication.

0:00-9:42
Play in

transcript

No transcript — this publisher did not publish one.

show notes

A test can score an agent’s last run. A feedback loop has to make the next run better.

Sam interviews Mark Pollack, founder of Spring AI, about Agent Control Theory: record the work, judge it against explicit requirements, measure the result and feed the evidence into the next attempt.

Pollack’s examples show why each part matters. A code-coverage prompt changed where an agent spent its effort without improving quality or cost. All 290 tests passed in a commonly used Spring PetClinic sample application. JetBrains developer Anton Arhipov wrote the feature specification, and his review missed a required lock order that an AI jury flagged and Pollack later confirmed. That established a lock-order violation, not a reproduced deadlock. A separate Agent Client Protocol SDK evaluation moved one cancellation requirement from failure to a passing rerun after an identifiable fix.

The loop does not eliminate judgment. Pollack says deterministic checks should come first, model judges should be grounded in evidence, and a human still has to catch a judge that is confidently measuring the wrong thing.

Sources

If you supervise coding agents, what evidence changed the next run rather than merely scoring the last one? Email SamEllisShow@protonmail.com. Suggested subject line: “The feedback loop caught this.” Anonymous notes and source-protection requests are welcome.

Send a source tip: SamEllisShow@protonmail.com.

How this episode was checked: an AI fact-checker and an AI editorial reviewer checked the script, and a human editor approved the exact script and the finished audio before publication.

links14

more episodes

All episodes