Skip to content
Sign in
Artwork for YPO Technology Network AI Brief
YPO Technology Network AI Brief · Today · 7 min

AI Agent Reliability: Test Wrong Decisions, Not Just Completed Tasks

Visa's new AI shopping benchmark found agents that failed usually still chose to buy. Stephen Forte explains why CEOs should test agents on the cases where the right answer is to decline or ask, and why memory kept outside the tool matters when a vendor update breaks. In this episode: Visa's Agentic Commerce Benchmark. Twenty-two models from twelve labs ran fifteen shopping scenarios three times each, for nine hundred and ninety runs. Overall pass rate fifty-seven point two percent; only thirty-four point two percent of model and scenario pairs passed all three attempts. Caveats that matter. Simulated market and customer, stopped before checkout, no money moved, a small scenario set, and Visa is an interested party. Read it for how it grades, not as a score for your business. How a run passes. An agent had to discover every customer requirement and then make the right call, including choosing nothing when nothing fits. Guessing the right product without asking the right questions could still fail. What failed agents did. In fifty-four percent of the failed runs, the model still chose to buy. That is half of the failures, not half of all runs. Visa also counted false refusals as failures. Completion is not correctness. Most agent pilots are graded on whether the task finished. Demos show the cases that go well. The cases that matter are where the right answer is no. Three tests before you trust an agent. Mix in decline-or-ask cases, rerun identical cases for repeatability, and score acting-when-it-shouldn't separately from refusing-when-it-should. Grok Bot update. A first-hand account of multi-step stalls, faster token burn, worse judgment and a false self-upgrade claim after a vendor update. Public context from Elon Musk and Matt Palmer on November eighth, without a named cause or timeline. Resilience outside the tool. Client work moved to Claude in about fifteen minutes because memory lived in backed-up ledgers. Keep a dependency inventory for tools with their own Linux machine. Retest agents after every vendor update. Sources: Visa, Nov 5: Agentic Commerce Benchmark release Visa whitepaper: Methodology and grading Elon Musk on X, Nov 8: Grok Bot fixes underway Matt Palmer on X, Nov 8: User feedback request xAI: Grok Bot product description The AI Brief from the YPO Technology Network is a daily executive briefing on the AI developments that matter to business leaders. Hosted by Stephen Forte.

0:00-7:39
Play in

transcript

No transcript — this publisher did not publish one.

show notes

Visa's new AI shopping benchmark found agents that failed usually still chose to buy. Stephen Forte explains why CEOs should test agents on the cases where the right answer is to decline or ask, and why memory kept outside the tool matters when a vendor update breaks.

In this episode:

  • Visa's Agentic Commerce Benchmark. Twenty-two models from twelve labs ran fifteen shopping scenarios three times each, for nine hundred and ninety runs. Overall pass rate fifty-seven point two percent; only thirty-four point two percent of model and scenario pairs passed all three attempts.
  • Caveats that matter. Simulated market and customer, stopped before checkout, no money moved, a small scenario set, and Visa is an interested party. Read it for how it grades, not as a score for your business.
  • How a run passes. An agent had to discover every customer requirement and then make the right call, including choosing nothing when nothing fits. Guessing the right product without asking the right questions could still fail.
  • What failed agents did. In fifty-four percent of the failed runs, the model still chose to buy. That is half of the failures, not half of all runs. Visa also counted false refusals as failures.
  • Completion is not correctness. Most agent pilots are graded on whether the task finished. Demos show the cases that go well. The cases that matter are where the right answer is no.
  • Three tests before you trust an agent. Mix in decline-or-ask cases, rerun identical cases for repeatability, and score acting-when-it-shouldn't separately from refusing-when-it-should.
  • Grok Bot update. A first-hand account of multi-step stalls, faster token burn, worse judgment and a false self-upgrade claim after a vendor update. Public context from Elon Musk and Matt Palmer on November eighth, without a named cause or timeline.
  • Resilience outside the tool. Client work moved to Claude in about fifteen minutes because memory lived in backed-up ledgers. Keep a dependency inventory for tools with their own Linux machine. Retest agents after every vendor update.

Sources:

The AI Brief from the YPO Technology Network is a daily executive briefing on the AI developments that matter to business leaders. Hosted by Stephen Forte.

links5

more episodes

All episodes