Free and Pro courses on LLM inference: vLLM, SGLang, KV cache, batching, quantization.
Markdown supported · ⌘↵ to post
The KV cache step is the one most people miss. Weights are a one-time number, but KV grows with every user and every token in the conversation, so a setup that fits on paper can run out of room the moment real traffic shows up. The 2 x 80 GB row makes that clear — fitting only at 0.95 utilization leaves no headroom at all.
For a small team the takeaway is even simpler: if the math says two GPUs minimum and four for comfortable serving, most small businesses shouldn't self-host a 70B at all. Start with a smaller model or a hosted option, and only move up when the workload proves you need it. The formula at the end is genuinely useful — you can redo it for any model instead of guessing from a spec sheet.
Ahmet Özel
The fit-or-no-fit conclusions are sensitive to the hardware capacity unit: the table treats an 80 GB GPU as decimal bytes before converting the usable fraction to GiB. I would make the device's reported memory in bytes an explicit calculator input rather than deriving it solely from the product label.
That is especially relevant for the two borderline rows, where a small capacity difference changes the conclusion. A worked deployment could show the reported device capacity, engine memory budget, and resulting cache allocation side by side. It would keep the useful weight and KV arithmetic reproducible on the exact hardware being considered.