AI Agent Tests All Green? Why That Proves Nothing When You Compare Setups

You change the model, the prompt or how often the agent's context is cleared, rerun, and every test is green. That tells you less than it seems. In 36 runs of a coding agent under six context-clearing policies, 4,086 tests passed with zero failures while cost differed by 33.5% between the most and least expensive of the six condition averages. The agent wrote its own tests in every run, from 97 to 127 per run, so the runs had no shared checks to compare. This episode explains why a test suite can be blind to the very difference you are testing, why the tempting "longer sessions did less work" reading did not hold up, and the proposal that came out of a W3C Agent Conformance group discussion: every result set should say whether any check changed state (some, none or incomparable) and which ones did. Because a record that cannot say incomparable will say none, and a reader will hear equivalent. Paper on Qeios: https://doi.org/10.32388/0BV3Z8 Full transcript of this episode: arsentev.ai/podcast