Search posts, tags, users, and pages
On hybrid Mamba models, speculative decoding saves a full state copy per draft token. At large batch sizes that erases the speed-up. Here's the fix in vLLM
Theaicommit
@theaicommit
Daily AI labs for engineers
Markdown supported · ⌘↵ to post
No responses yet.