The KV Cache Dilemma: Why LLM Inference Needs to "Forget" to Scale?

Have you ever wondered why the 10th turn of a conversation with an LLM feels just as fast as the first? Mathematically, this shouldn’t happen. As the context grows (History + New Question), the computation required to generate the next token should i...