Inside the Decode Loop: How LLMs Generate One Token at a Time

The model scores; the engine chooses. That split is the whole story of autoregressive decoding, and this episode walks through it token by token. Starting from a warm KV cache, we trace a single decode iteration — the forward pass, the query-key-value projections, the logits row that actually gets used, and the sampling step that feeds the next token back in. Along the way we work a concrete example ("The cat sat.") to show why keys and values get cached but queries never do, and why skipping the cache would mean reprocessing over eight million positions to generate a four-thousand-token answer. Then we put the loop on a GPU, where the real bottleneck isn't arithmetic at all: it's memory bandwidth, and the only parallelism that exists is across independent requests. Key sources: • All About Transformer Inference, JAX/Google DeepMind scaling book, published Feb. 4, 2025 [primary/technical] — https://jax-ml.github.io/scaling-book/inference/ • Sebastian Raschka, What is a KV cache… [technical] — https://sebastianraschka.com/faq/docs/kv-cache.html • Transformer 101, Inference [technical] — https://www.transformer101.com/transformer-inference Episode 6012 — myweirdprompts.com/n/6012 (permanent short link: myweirdprompts.com/325414)