8. A Baseline for Judging the Agent's Setup

How to build a small baseline of real tasks with known answers, and use it to choose a Claude model and effort level by the total cost of finished work, including your review and repair time. It closes with recent Claude Code changes that affect usage warnings, sandboxed browser tests, headless sign-in and review. This episode is narrated by AI-generated synthetic voices. Episode page & show notes Visit website A fixed set of tasks and a record of results You can't judge a change to your agent setup by feel. Runs vary, and this week's tasks differ from last week's. What you need is a fixed set of real tasks and the results you got on them before. Cost here means the cost of a finished change: the token bill plus your minutes reviewing and repairing the diff. The chapter builds three tasks from the shared-list app's own history: the empty-name validation rule, the fast-add bug fixed from a reproduction note, and the done checkbox that needed a migration. Each task folder holds a starting commit hash, the exact request text, testable success criteria, and the check that settles the task. The regression tests you wrote when fixing each bug go in as hidden checks the agent never sees, following Anthropic's guidance on testing agents. Each run is replayed in a clean Git worktree with today's CLAUDE.md and settings copied in. Runs avoid the bare option, because bare mode skips the CLAUDE.md, skills and hooks being tested. Two dials: model and effort Sonnet, Opus, Haiku and the opusplan alias are compared, along with the reasoning-effort levels. The chapter shows how to set both with /model, /effort, command-line options, and the model and effort keys in the settings files, including which file wins when they disagree. /usage, /cost and /context report usage. A run that hits a usage limit counts as interrupted, not failed, and gets rerun. When the cheaper setup costs more In an illustrative comparison, which the chapter says is not a measured claim, each task runs twice under Opus at high effort and twice under Sonnet at medium. Sonnet matches Opus on the validation rule and the bug fix. On one done-flag run, Sonnet passes the suite but fails the clicked journey: the database client never writes the done column. Once you price your time, that fifteen-minute repair outweighs the token savings. The result is a default per kind of task: Sonnet at medium in settings, and a CLAUDE.md rule sending database work to Opus at high effort. The chapter names three ways to fool yourself: comparing single runs, writing criteria vague enough that everything passes, and counting tokens while ignoring review time. Recent Claude Code updates Versions 2.1.303 through 2.1.310 are listed in the Claude Code changelog. Version 304 warns Pro and Max users before resuming a large session. 305 adds a Linux sandbox setting that lets headless Chromium run. 306 fixes sign-in failures in claude -p runs. 309 makes ultrareview require explicit workspace trust. A new API version removes temperature and sampling settings and changes how keys are sent, which matters only if you code against the API directly.