Understanding KV-Cache Quantization by Building It in GPT-2

Large language models rely on a KV cache to avoid recomputing attention during autoregressive generation, but this cache grows linearly with sequence length and quickly becomes a memory bottleneck. In