Skip to content
Sign in
Artwork for LLM.co
LLM.co · Sunday · 5 min

How Many GPUs Do You Actually Need? A Practical Sizing Guide

GPU sizing for on-premises AI rarely starts with the right question. Vendors lead with reference architectures built for worst-case scenarios, and organizations end up over-provisioned before a single workload requirement has been defined. This episode of LLM.co works through a practical, math-first approach to figuring out how much GPU memory and how many cards a real inference workload actually needs — drawing on the practical GPU sizing guide from hardware.co as its foundation. The episode walks through each layer of the sizing calculation in sequence, from model weights to concurrency planning, covering: Weight memory as the baseline: why the two-gigabytes-per-billion-parameter rule in FP16 is your starting point, and how quantization (FP8, INT4) can cut that figure dramatically before you ever consider adding hardware. The VRAM floor vs. ceiling distinction: weights only occupy 60–70% of usable VRAM in a well-planned deployment — the rest goes to KV cache, framework overhead, and serving headroom. Why context length is the most common sizing mistake: at long contexts and high concurrency, the KV cache can dwarf model weights entirely, and three software-side choices (grouped-query attention, capped context, paged attention) address it before any hardware purchase. Separating throughput from latency: aggregate tokens-per-second and per-user time-to-first-token are different constraints that drive different decisions — and the episode shows how to multiply through both to arrive at a defensible GPU count. Memory bandwidth over TFLOPS: why LLM inference is memory-bandwidth-bound, and what that means for reading GPU spec sheets. Sizing to the second-worst normal hour, not the theoretical peak: reference architectures assume your worst day is every day; real workloads — whether bursty document review or steady clinical summarization — are not shaped that way. The five numbers to write down before requesting any quote — target model and quantization, p95 token lengths, peak concurrent users, acceptable token rate, and time-to-first-token — are the same inputs that inform decisions about comparing AI server configurations across different deployment sizes. Teams exploring on-premises deployment may also want to look at fine-tuning and model training services, where GPU memory requirements shift significantly from inference workloads. For more on keeping production LLM deployments stable during infrastructure changes, the episode How to Replace a Public LLM API Without Breaking Production covers the operational side of that transition. hardware.co

0:00-5:14
Play in

transcript

No transcript — this publisher did not publish one.

show notes

GPU sizing for on-premises AI rarely starts with the right question. Vendors lead with reference architectures built for worst-case scenarios, and organizations end up over-provisioned before a single workload requirement has been defined. This episode of LLM.co works through a practical, math-first approach to figuring out how much GPU memory and how many cards a real inference workload actually needs — drawing on the practical GPU sizing guide from hardware.co as its foundation.

The episode walks through each layer of the sizing calculation in sequence, from model weights to concurrency planning, covering:

  • Weight memory as the baseline: why the two-gigabytes-per-billion-parameter rule in FP16 is your starting point, and how quantization (FP8, INT4) can cut that figure dramatically before you ever consider adding hardware.
  • The VRAM floor vs. ceiling distinction: weights only occupy 60–70% of usable VRAM in a well-planned deployment — the rest goes to KV cache, framework overhead, and serving headroom.
  • Why context length is the most common sizing mistake: at long contexts and high concurrency, the KV cache can dwarf model weights entirely, and three software-side choices (grouped-query attention, capped context, paged attention) address it before any hardware purchase.
  • Separating throughput from latency: aggregate tokens-per-second and per-user time-to-first-token are different constraints that drive different decisions — and the episode shows how to multiply through both to arrive at a defensible GPU count.
  • Memory bandwidth over TFLOPS: why LLM inference is memory-bandwidth-bound, and what that means for reading GPU spec sheets.
  • Sizing to the second-worst normal hour, not the theoretical peak: reference architectures assume your worst day is every day; real workloads — whether bursty document review or steady clinical summarization — are not shaped that way.

The five numbers to write down before requesting any quote — target model and quantization, p95 token lengths, peak concurrent users, acceptable token rate, and time-to-first-token — are the same inputs that inform decisions about comparing AI server configurations across different deployment sizes. Teams exploring on-premises deployment may also want to look at fine-tuning and model training services, where GPU memory requirements shift significantly from inference workloads. For more on keeping production LLM deployments stable during infrastructure changes, the episode How to Replace a Public LLM API Without Breaking Production covers the operational side of that transition.

hardware.co

links5

more episodes

All episodes