Prefill vs decode: why LLM inference is memory-bandwidth boundAsk why an LLM streams its answer at a steady ~100 tokens/s even on a GPU rated for nearly a petaFLOP, and you end up at one idea: generation has two phases with very different bottlenecks. Two phases110h ago · 4 min readinferenceacademy.hashnode.dev
KV cache math: why Llama 3.1 8B needs ~128 KB per tokenIf you serve LLMs, the KV cache, not the weights, usually decides how many users fit on one GPU. The good news is that its size is easy to calculate from config.json, and the calculation explains a loOct 28 · 4 min readinferenceacademy.hashnode.dev