16h ago · 8 min read · A practical observability setup for LLM inference on KServe — and the one-line misconfiguration it caught. LLM serving breaks the assumptions behind ordinary service dashboards. A single "request late
Join discussionMay 9 · 10 min read · Every production LLM deployment using speculative decoding is likely running a fixed speculation length of γ=4. That number comes from early benchmarks, it has been copy-pasted across blog posts and framework defaults, and almost nobody questions it....
Join discussion
May 6 · 10 min read · TL;DR — I was happily running Qwen3.6 on llama.cpp. Then I saw claims of 2× speed with vLLM + NVFP4 + DFlash. So I installed it, fought through crashes, and measured it myself. Verdict: it's real. 88–
Join discussionMay 3 · 5 min read · The Windows local LLM story just got interesting. Someone recently demonstrated Qwen3's 27B model running at 72 tokens per second on an RTX 3090 — natively on Windows. No WSL. No Docker. Just a portable vLLM launcher. If you've been running local mod...
Join discussion
Apr 28 · 11 min read · Mixture-of-Experts models have dominated the open-weight frontier in 2026. Llama 4 Scout (17B-16E), Llama 4 Maverick (17B-128E), DeepSeek V4-Pro (1.6T-49B active), and Qwen3.6-Plus all use sparse expert routing to scale parameters without proportiona...
Join discussion
Apr 22 · 11 min read · Serving a large language model in production is a solved problem — until your traffic doubles, your structured output pipeline slows to a crawl, or your cloud bill arrives. The choice of inference engine determines how many GPUs you actually need, ho...
Join discussion
