---
title: "Discussion on \"Why Speculative Decoding Slows Hybrid Mamba Models at Large Batch Sizes\" | Hashnode"
description: "Discussion on \"Why Speculative Decoding Slows Hybrid Mamba Models at Large Batch Sizes\". On hybrid Mamba models, speculative decoding saves a full state copy per draft token. At large batch sizes that erases the speed-up. Here's the fix in vLLM"
url: "https://hashnode.com/posts/why-speculative-decoding-slows-hybrid-mamba-models-at-large-batch-sizes/6adaa1e85e0fca8687b12dff"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Why Speculative Decoding Slows Hybrid Mamba Models at Large Batch Sizes](https://theaicommit.hashnode.dev/why-speculative-decoding-slows-hybrid-mamba-models-at-large-batch-sizes?utm_source=hashnode&utm_medium=feed)

On hybrid Mamba models, speculative decoding saves a full state copy per draft token. At large batch sizes that erases the speed-up. Here's the fix in vLLM

Oct 22 · 9 min read · theaicommit.hashnode.dev  
Read the article

-   [#llm-engineering-ai-infrastructure-python-gpu-optimization-performance-engineering-machine-learning-systems-devops-artificial-intelligence-software-engineering-programming-cloud-computing-deep-learning](https://hashnode.com/tag/llm-engineering-ai-infrastructure-python-gpu-optimization-performance-engineering-machine-learning-systems-devops-artificial-intelligence-software-engineering-programming-cloud-computing-deep-learning)

## Written by

[Theaicommit](https://hashnode.com/@theaicommit)

@theaicommit

  
[View profile](https://hashnode.com/@theaicommit)

Daily AI labs for engineers

## Responses

Markdown supported · ⌘↵ to post

No responses yet.
