Inside LLM Inference: Prefill, Decode, and the KV Cache

A single request to an LLM isn't one computation—it's a stack of stages wearing a trench coat. This episode walks the full pipeline: flattening a chat transcript into one flat document with special role tokens, tokenizing text into IDs, the massively parallel prefill pass that produces your first token, and the strictly sequential decode loop that produces every token after. Along the way: why forgetting one flag called add_generation_prompt makes the model write your half of the conversation, why "encoding" and "decoding" each mean two different things, and how a KV cache can quietly outgrow the model weights it's serving. If you've ever wondered where your time-to-first-token actually goes, this is the tour. Key sources: • Nick Gustafson, What an Inference Engine Actually Does — https://thegustafson.com/blog/what-an-inference-engine-does • BentoML, How does LLM inference work? | LLM Inference Handbook — https://bentoml.com/llm/llm-inference-basics/how-does-llm-inference-work • JAX-ML / Google DeepMind, All About Transformer Inference | How To Scale Your Model, published Feb 4 2025 — https://jax-ml.github.io/scaling-book/inference/ Episode 6011 — myweirdprompts.com/n/6011 (permanent short link: myweirdprompts.com/350177)