The 283 → 1278 tok/s jump is a nice demonstration, but it's worth noting the cache isn't free: memory grows linearly with sequence length × layers × heads, and with torch.cat on every step you also reallocate the whole tensor each token. Preallocating a fixed-size buffer and writing into a slice (or moving to paged blocks like vLLM) would likely widen the gap further. Did you measure how the speedup changes as context length grows, and where GPU memory becomes the limit?