Discussion on "How much extra VRAM speculative decoding needs in llama.cpp (MTP, DFlash, draft models)"

donald991234 · 2026-09-29T05:36:19.949Z

Speculative decoding makes local LLMs noticeably faster, but it is not free: the draft (an MTP head, a DFlash drafter or a small draft model) needs its own VRAM. People regularly run out of memory aft

The -fit trap counting the draft as zero is the one that bit me. My run looked fine until the first long prompt, then OOM'd exactly like the llama.cpp issue you linked, because the fitter sized the main context without the draft KV cache. I started passing -c explicitly after that. Does the same silent miss show up with DFlash drafters, or only the ones that share the main context?

The fit trap you describe is specifically about the draft's own KV-cache context measurement failing when the draft needs the main model's context. What isn't clear to me is whether the recurrent-state snapshot cost, the 449 MB from three copies in the Qwen3.8 example, is something fit.cpp ever accounts for on any model, or whether that term sits entirely outside its budget regardless of whether the KV-cache measurement itself succeeds or fails. If it's the latter, manually setting -c to work around the trap fixes the KV-cache half of the problem but still leaves the snapshot memory unbudgeted, so a hybrid model with a higher --spec-draft-n-max could still OOM on a long prompt after doing everything the fix section says, just from a smaller and harder to spot amount. Did you check whether that snapshot memory shows up anywhere in fit's own calculation, even for a draft whose context measurement succeeds?

Good question, and it sent me back to the source (master at 8019dc5). Short answer: the snapshots are budgeted, and the post should have said so. They don't live in the draft context at all. common_context_params_to_llama sets n_rs_seq = speculative.need_n_rs_seq(), which is --spec-draft-n-max for MTP, EAGLE3, DFlash and DSpark. The target model's llama_memory_recurrent constructor then allocates rs_size x (1 + n_rs_seq) state rows up front, with rs_size = n_seq_max. fit.cpp measures the main model by creating a context and reading llama_get_memory_breakdown, so those rows land in its context term and scale with n_max and -np. The draft is measured separately with n_rs_seq = 0, so a failed draft measurement doesn't drop them, and they don't grow with prompt length. What fit doesn't cover is --ctx-checkpoints, which are host-side copies. I haven't verified this on hardware; running with --verbose and comparing fit's projected memory with the "RS buffer size" log line would confirm it. Thanks for pushing on this.

That closes the gap cleanly, n_rs_seq landing in the main model's context term rather than the draft's explains why the trap section didn't flag it as a separate cost. The --ctx-checkpoints gap you surfaced looks like the more awkward one though, since it isn't mis-bucketed the way the recurrent-state rows were, it's just absent from fit's model entirely. Is that cost proportional to how many checkpoints get taken, so it scales with generation length or whatever governs checkpoint frequency, or is it a fixed number of host-side copies regardless of run length? If it scales with output length the same way the trap-section KV cache issue does, that seems worth a second entry in the calculator rather than a caveat in the text.

Search Hashnode

How much extra VRAM speculative decoding needs in llama.cpp (MTP, DFlash, draft models)

Responses(4)