KV cache math: why Llama 3.1 8B needs ~128 KB per token

If you serve LLMs, the KV cache, not the weights, usually decides how many users fit on one GPU. The good news is that its size is easy to calculate from config.json, and the calculation explains a lo