TTheaicommit·Oct 22Why Speculative Decoding Slows Hybrid Mamba Models at Large Batch SizesOn hybrid Mamba models, speculative decoding saves a full state copy per draft token. At large batch sizes that erases the speed-up. Here's the fix in vLLM9 min read·theaicommit.hashnode.dev
TTheaicommit·Oct 22How Sorting Your Prompts Speeds Up a Mixture-of-Experts Batch JobSimilar prompts pick the same experts, so a sorted batch reads fewer weights per step. Hugging Face measured 5–17% more throughput on unchanged vLLM.9 min read·theaicommit.hashnode.dev
AAhmed Adawy·Oct 2LLM Inference Engineering: Overcoming the KV-Cache Bottleneck and Maximizing Production ThroughputLLM Engineering AI Infrastructure Python GPU Optimization Performance Engineering Machine https://leanpub.com/b/thegenerativeaillmengineeringbundle3 min read·adawy.hashnode.dev