Thank you for your breakdown! While this article was just a projection of my current knowledge, I’ll definitely highlight this distinction in a future update. Thanks for adding so much value to the post!"
Today, I got to learn something new! 'The distinction between the prefill phase (compute-bound, parallelized prompt processing) and the decoding phase (memory-bandwidth bound, memory-bound autoregressive token generation) is where so many benchmarking comparisons fall flat'. Looking forward to learn more.
Putting RoPE and the KV cache family in one article makes sense because they are the two things that decide what long context actually costs you. One framing worth adding for readers coming from the serving side: MQA and GQA are memory bandwidth optimisations more than compute ones. Decoding is bandwidth-bound - you re-read the whole cache every token - so shrinking the number of KV heads speeds up generation even when FLOPs barely move, which surprises people who benchmark on prefill and see nothing. The practical consequence of the cache section is that serving economics stop being about tokens per second and start being about how many contexts fit in memory at once. Long-running agent workloads keep large caches resident across many steps, so cost per resident context-hour predicts your bill better than cost per token does. Worth noting on RoPE too that extending context at inference is not free - the positions the model never trained on behave differently, which is why scaling tricks exist and why long-context benchmarks disagree with each other so much.