---
title: "Comment by AI Developer on \"KV Cache Optimisation in AI Inference\" | Hashnode"
description: "The 283 → 1278 tok/s jump is a nice demonstration, but it's worth noting the cache isn't free: memory grows linearly with sequence length × layers × heads, and"
url: "https://hashnode.com/posts/kv-cache-optimisation-in-ai-inference/6adf5f5e5bb6fa9a24a51c06/comment/6adf88bf2263bbe898e12569"
---

[Discussion](https://hashnode.com/posts/kv-cache-optimisation-in-ai-inference/6adf5f5e5bb6fa9a24a51c06)

Comment [KV Cache Optimisation in AI Inference](https://hashnode.com/posts/kv-cache-optimisation-in-ai-inference/6adf5f5e5bb6fa9a24a51c06)

[AI Developer](https://hashnode.com/@lycore)

[Oct 26](https://hashnode.com/posts/kv-cache-optimisation-in-ai-inference/6adf5f5e5bb6fa9a24a51c06/comment/6adf88bf2263bbe898e12569)

The 283 → 1278 tok/s jump is a nice demonstration, but it's worth noting the cache isn't free: memory grows linearly with sequence length × layers × heads, and with torch.cat on every step you also reallocate the whole tensor each token. Preallocating a fixed-size buffer and writing into a slice (or moving to paged blocks like vLLM) would likely widen the gap further. Did you measure how the speedup changes as context length grows, and where GPU memory becomes the limit?
