Unharness Your Agents

Dan and John Berryman discuss why today’s terminal- and IDE-centered agent harnesses are too narrow. John argues that agents should be able to see and act across the applications, websites, files, and physical environments that make up a person’s life. His Rook project is an attempt to make those contexts addressable while allowing people to keep using the agent harnesses they already trust. The conversation turns to a future in which websites expose agent-facing capabilities and applications reshape themselves around conversations. John imagines an assistant that can combine a person’s notes and shopping list with location, store inventory, and aisle information. He says feasibility is the major barrier that recently moved into reach, while security, transparency, trust, and standardization remain unresolved. That tension becomes concrete in the discussion of permissions and sandboxing. John expects people to begin with constrained, read-only access and gradually grant more authority, with dry runs, approvals, and reversibility helping determine when an agent can act on its own. He also points to emerging conventions such as skills files, AGENTS.md, and llms.txt as ways for agents to discover what they can do in a given domain. John’s strongest practical advice is to replace bespoke workflow code with skills written in plain English whenever the model is capable enough to follow the instructions. He describes building a Zoom assistant and a candidate-vetting workflow this way, arguing that subject-matter experts may eventually be able to read and rewrite the software directly. In his view, the agent is increasingly just a loop around a model and tools, with frameworks such as LangGraph becoming less necessary for many applications. The discussion of memory is more skeptical. John argues that chunking old conversations and retrieving them by textual similarity does not reproduce the way human experience turns mistakes into procedural knowledge and taste. For now, he prefers explicit, visible review of a completed task followed by packaging the generalized process as a skill. That approach is also his answer to the limits of AI-generated writing: agents can remove ums, edit dead space, and produce polished drafts, but they still do not know what a particular person considers good. Looking ahead, John expects more agentic applications, just-in-time custom software, and world models that simulate environments for robotics. He is optimistic but clear-eyed about the risks, closing with a plea to direct the technology toward useful and humane outcomes rather than dystopia. Full episode notes Click here to view the episode transcript. Chapters (00:00) - Why agents should be unharnessed (03:32) - Rook and context-aware agents (13:23) - Agent-facing applications and trust (18:23) - Sandboxing, dry runs, and reversibility (21:29) - The new programming language is English (30:28) - Why bespoke agent frameworks are fading (34:13) - What agent memory gets wrong (42:50) - Explicit memory through skills (51:07) - Taste as the final frontier (57:49) - World models and the future ⠀ Links from the show -------------------- Arcturus Labs Blog The AI Product Era You're Building For Might Already Be Over Unharnessed Agents Power the Future of AI Products Relevant Search Prompt Engineering for LLMs Agent Client Protocol OpenClaw Hermes Pi LangGraph Pydantic AI Tailscale OpenStreetMap Model Context Protocol ⠀ Guests ------- John Berryman, Founder, Arcturus Labs Website LinkedIn X YouTube ⠀ Follow the podcast ------------------- LinkedIn Threads Instagram TikTok ⠀ Follow Dan Gerlanc ------------------- X LinkedIn Threads Bluesky