Prefill vs decode: why LLM inference is memory-bandwidth bound

Ask why an LLM streams its answer at a steady ~100 tokens/s even on a GPU rated for nearly a petaFLOP, and you end up at one idea: generation has two phases with very different bottlenecks. Two phases