030 - How AI Agents Work: Harnesses, Memory, Tools and Loops (Deep Dive)

How does an AI agent go from a language model that generates text to a system that can research, use software, change files, and work through a task? Episode 30 explains the core architecture behind modern AI agents, focusing on the components surrounding the model rather than benchmark scores. The episode starts with the LLM and the harness. The model provides the language and reasoning capability, while the harness supplies instructions, context, memory, tools, state, permissions, and execution behavior. OpenAI's current Agents SDK includes orchestration, tools, guardrails, and tracing, while its newer agent infrastructure supports controlled file and command execution. Anthropic's work on long-running agents similarly emphasizes context management and durable artifacts for work that spans multiple sessions. Memory is then separated from the model's immediate context. The episode explains working context alongside longer-lived procedural, semantic, and episodic information, while noting that implementations vary. This leads into retrieval-augmented generation, or RAG, and the broader idea of context engineering. RAG can use embeddings and vector search, but an agent can also retrieve information through SQL, document search, APIs, files, or web tools. The episode emphasizes just-in-time retrieval because sending too much information into a context window can make useful information harder to use. Context engineering ties these pieces together. The challenge is not simply putting more information into the model. It is selecting useful information at the right time and in a form the model can use. That includes deciding which memories to retrieve, which tool results to keep, which instructions to prioritize, and what older information can be compacted or discarded. Tracing provides visibility into the run. A trace can record model calls, retrieval, tool calls, inputs and outputs, timing, token usage, errors, and other metadata. Langfuse documents traces as structured records containing observations such as generations, retrieval operations, and tool calls, while OpenAI's Agents SDK includes tracing and observability for agent workflow execution. The final concept is LLM Ops: the ongoing engineering loop around the running system. Teams inspect traces, identify failures or inefficiencies, adjust prompts, tools, retrieval, models, memory, or guardrails, and run the system again. The episode closes with a simple architecture: LLM, harness, context and memory, retrieval, tools, loop, guardrails, tracing, and operational improvement. References OpenAI — New tools for building agentshttps://openai.com/index/new-tools-for-building-agents/ OpenAI — The next evolution of the Agents SDKhttps://openai.com/index/the-next-evolution-of-the-agents-sdk/ Anthropic — Building Effective AI Agentshttps://resources.anthropic.com/building-effective-ai-agents Anthropic — Effective context engineering for AI agentshttps://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents Anthropic — Effective harnesses for long-running agentshttps://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents Anthropic — Writing effective tools for AI agentshttps://www.anthropic.com/engineering/writing-tools-for-agents Langfuse — Core Concepts: Traces and Observationshttps://langfuse.com/docs/observability/data-model Langfuse — Tracinghttps://langfuse.com/academy/tracing Voice narration is AI-generated.