Running LLMs is easy. Scaling Inference on Kubernetes is Hard!

Running an open-source LLM locally is increasingly easy. Running that same model efficiently for hundreds or thousands of users is an entirely different engineering problem. In this episode of Kubernetes Bytes, Bhavin sits down with Cedric Clyburn from Red Hat to unpack what actually happens when organizations move from experimenting with local models to operating AI inference at scale on Kubernetes. They discuss why traditional application metrics don’t fully describe LLM performance and introduce the three important inference metrics: time to first token, inter-token latency, and throughput. Cedric also explains what inference servers such as vLLM actually do, how KV cache consumes GPU memory, why agentic workloads dramatically increase inference demand, and how optimizations such as continuous batching and paged attention help make expensive GPUs more efficient. The conversation goes deeper into GPU architecture and memory movement, including HBM, SRAM, prefill and decode, before exploring model compression and quantization as another way to reduce infrastructure requirements. Finally, they discuss llm-d and the emerging Kubernetes-native distributed inference stack, including KV-cache-aware routing, Day 2 model operations, model upgrades, and why AI sovereignty and compliance may drive more organizations toward self-hosted inference. Topics include: Hosted AI APIs vs. running your own models AI inference economics Time to first token (TTFT) Inter-token latency (ITL) Token throughput Why inference servers exist vLLM and SGLang KV cache GPU memory and HBM Prefill vs. decode Continuous batching Paged attention Distributed inference Model compression and quantization LLM evaluation and benchmarking vLLM vs. llm-d Kubernetes inference-aware routing Day 2 operations for AI models AI sovereignty and private inference Running smaller models locally on CPUs and edge devices If you’re a Kubernetes, platform engineering, infrastructure, or AI engineering practitioner trying to understand how production LLM serving actually works, this episode provides a practical introduction to the infrastructure underneath modern AI applications. Show Notes: Cedric Clyburn - https://www.linkedin.com/in/cedricclyburn/ IBM Technology Youtube - https://www.youtube.com/@IBMTechnology https://github.com/vllm-project/vllm https://llm-d.ai/ https://github.com/kserve/kserve https://gateway-api.sigs.k8s.io/ https://github.com/vllm-project/llm-compressor https://github.com/vllm-project/guidellm