---
title: "Discussion on \"KV Cache Optimisation in AI Inference\" | Hashnode"
description: "Discussion on \"KV Cache Optimisation in AI Inference\". Learn how the KV cache speeds up LLM inference. We build a small GPT in PyTorch and benchmark cached vs uncached generation: a 4.5x speedup."
url: "https://hashnode.com/posts/kv-cache-optimisation-in-ai-inference/6adf5f5e5bb6fa9a24a51c06"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [KV Cache Optimisation in AI Inference](https://ai-inference-pratyush.hashnode.dev/kv-cache-optimisation-in-ai-inference?utm_source=hashnode&utm_medium=feed)

Learn how the KV cache speeds up LLM inference. We build a small GPT in PyTorch and benchmark cached vs uncached generation: a 4.5x speedup.

Oct 26 · 8 min read · ai-inference-pratyush.hashnode.dev  
Read the article

-   [#inference](https://hashnode.com/tag/inference)
-   [#ai](https://hashnode.com/tag/ai)
-   [#llm](https://hashnode.com/tag/llm)
-   [#kv-cache](https://hashnode.com/tag/kv-cache)
-   [#gpt](https://hashnode.com/tag/gpt)

## Written by

[Pratyush Ranjan](https://hashnode.com/@pratyushranjan02)

@pratyushranjan02 · Bengaluru, India

  
[View profile](https://hashnode.com/@pratyushranjan02)

Sharing interesting engineering stuff

Software engineer. Eager to learn storage engines, distributed databases, AI inference

## Responses 1

Markdown supported · ⌘↵ to post

[AI Developer](https://hashnode.com/@lycore)

[Oct 26](https://hashnode.com/posts/kv-cache-optimisation-in-ai-inference/6adf5f5e5bb6fa9a24a51c06/comment/6adf88bf2263bbe898e12569)

The 283 → 1278 tok/s jump is a nice demonstration, but it's worth noting the cache isn't free: memory grows linearly with sequence length × layers × heads, and with torch.cat on every step you also reallocate the whole tensor each token. Preallocating a fixed-size buffer and writing into a slice (or moving to paged blocks like vLLM) would likely widen the gap further. Did you measure how the speedup changes as context length grows, and where GPU memory becomes the limit?
