---
title: "Discussion on \"How much extra VRAM speculative decoding needs in llama.cpp (MTP, DFlash, draft models)\" | Hashnode"
description: "Discussion on \"How much extra VRAM speculative decoding needs in llama.cpp (MTP, DFlash, draft models)\". Speculative decoding makes local LLMs noticeably faster, but it is not free: the draft (an MTP head, a DFlash drafter or a small draft model) needs its own VRAM. People regularly run out of memory aft"
url: "https://hashnode.com/posts/how-much-extra-vram-speculative-decoding-needs-in-llama-cpp-mtp-dflash-draft-models/6abb4e53f601645d75bc6149"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [How much extra VRAM speculative decoding needs in llama.cpp (MTP, DFlash, draft models)](https://modelvram.hashnode.dev/how-much-extra-vram-speculative-decoding-needs-in-llama-cpp-mtp-dflash-draft-models?utm_source=hashnode&utm_medium=feed)

Speculative decoding makes local LLMs noticeably faster, but it is not free: the draft (an MTP head, a DFlash drafter or a small draft model) needs its own VRAM. People regularly run out of memory aft

Sep 29 · 4 min read · modelvram.hashnode.dev  
Read the article

-   [#llm](https://hashnode.com/tag/llm)
-   [#gpu](https://hashnode.com/tag/gpu)
-   [#machine-learning](https://hashnode.com/tag/machine-learning)

## Written by

[donald991234](https://hashnode.com/@donald991234)

@donald991234

  
[View profile](https://hashnode.com/@donald991234)

## Responses 4

Markdown supported · ⌘↵ to post

[Kartik N V J K](https://hashnode.com/@kartik-nvj)

[Sep 29](https://hashnode.com/posts/how-much-extra-vram-speculative-decoding-needs-in-llama-cpp-mtp-dflash-draft-models/6abb4e53f601645d75bc6149/comment/6abc1575b9afa6c19cae8ba7)

The -fit trap counting the draft as zero is the one that bit me. My run looked fine until the first long prompt, then OOM'd exactly like the llama.cpp issue you linked, because the fitter sized the main context without the draft KV cache. I started passing -c explicitly after that. Does the same silent miss show up with DFlash drafters, or only the ones that share the main context?

[Mihai\_LeanZero](https://hashnode.com/@mihaileanzero)

[Sep 29](https://hashnode.com/posts/how-much-extra-vram-speculative-decoding-needs-in-llama-cpp-mtp-dflash-draft-models/6abb4e53f601645d75bc6149/comment/6abb8b20126a47d11ee0a643)

The fit trap you describe is specifically about the draft's own KV-cache context measurement failing when the draft needs the main model's context. What isn't clear to me is whether the recurrent-state snapshot cost, the 449 MB from three copies in the Qwen3.8 example, is something fit.cpp ever accounts for on any model, or whether that term sits entirely outside its budget regardless of whether the KV-cache measurement itself succeeds or fails. If it's the latter, manually setting -c to work around the trap fixes the KV-cache half of the problem but still leaves the snapshot memory unbudgeted, so a hybrid model with a higher --spec-draft-n-max could still OOM on a long prompt after doing everything the fix section says, just from a smaller and harder to spot amount. Did you check whether that snapshot memory shows up anywhere in fit's own calculation, even for a draft whose context measurement succeeds?

[donald991234](https://hashnode.com/@donald991234)

[Sep 29](https://hashnode.com/posts/how-much-extra-vram-speculative-decoding-needs-in-llama-cpp-mtp-dflash-draft-models/6abb4e53f601645d75bc6149/comment/6abb8b20126a47d11ee0a643#reply-6abb9986ce57b82d0614f168)

Good question, and it sent me back to the source (master at 8019dc5). Short answer: the snapshots are budgeted, and the post should have said so. They don't live in the draft context at all. common\_context\_params\_to\_llama sets n\_rs\_seq = speculative.need\_n\_rs\_seq(), which is --spec-draft-n-max for MTP, EAGLE3, DFlash and DSpark. The target model's llama\_memory\_recurrent constructor then allocates rs\_size x (1 + n\_rs\_seq) state rows up front, with rs\_size = n\_seq\_max. fit.cpp measures the main model by creating a context and reading llama\_get\_memory\_breakdown, so those rows land in its context term and scale with n\_max and -np. The draft is measured separately with n\_rs\_seq = 0, so a failed draft measurement doesn't drop them, and they don't grow with prompt length. What fit doesn't cover is --ctx-checkpoints, which are host-side copies. I haven't verified this on hardware; running with --verbose and comparing fit's projected memory with the "RS buffer size" log line would confirm it. Thanks for pushing on this.

[Mihai\_LeanZero](https://hashnode.com/@mihaileanzero)

[Sep 30](https://hashnode.com/posts/how-much-extra-vram-speculative-decoding-needs-in-llama-cpp-mtp-dflash-draft-models/6abb4e53f601645d75bc6149/comment/6abb8b20126a47d11ee0a643#reply-6abcd59b4c2c6a1d4ac96435)

That closes the gap cleanly, n\_rs\_seq landing in the main model's context term rather than the draft's explains why the trap section didn't flag it as a separate cost. The --ctx-checkpoints gap you surfaced looks like the more awkward one though, since it isn't mis-bucketed the way the recurrent-state rows were, it's just absent from fit's model entirely. Is that cost proportional to how many checkpoints get taken, so it scales with generation length or whatever governs checkpoint frequency, or is it a fixed number of host-side copies regardless of run length? If it scales with output length the same way the trap-section KV cache issue does, that seems worth a second entry in the calculator rather than a caveat in the text.
