---
title: "Comment by Steven Mock on \"How much GPU memory do you need to serve a 70B model?\" | Hashnode"
description: "The KV cache step is the one most people miss. Weights are a one-time number, but KV grows with every user and every token in the conversation, so a setup that"
url: "https://hashnode.com/posts/how-much-gpu-memory-do-you-need-to-serve-a-70b-model/6aeebac364fc9b5ba49982ec/comment/6aeec7c15c8154293b3ac4b1"
---

[Discussion](https://hashnode.com/posts/how-much-gpu-memory-do-you-need-to-serve-a-70b-model/6aeebac364fc9b5ba49982ec)

Comment [How much GPU memory do you need to serve a 70B model?](https://hashnode.com/posts/how-much-gpu-memory-do-you-need-to-serve-a-70b-model/6aeebac364fc9b5ba49982ec)

[Steven Mock](https://hashnode.com/@stevenmock)

[5d ago](https://hashnode.com/posts/how-much-gpu-memory-do-you-need-to-serve-a-70b-model/6aeebac364fc9b5ba49982ec/comment/6aeec7c15c8154293b3ac4b1)

The KV cache step is the one most people miss. Weights are a one-time number, but KV grows with every user and every token in the conversation, so a setup that fits on paper can run out of room the moment real traffic shows up. The 2 x 80 GB row makes that clear — fitting only at 0.95 utilization leaves no headroom at all.

For a small team the takeaway is even simpler: if the math says two GPUs minimum and four for comfortable serving, most small businesses shouldn't self-host a 70B at all. Start with a smaller model or a hosted option, and only move up when the workload proves you need it. The formula at the end is genuinely useful — you can redo it for any model instead of guessing from a spec sheet.