Tag feed

#vllm

42 posts1 followers

Explore Hashnode

Alternatives

Trending tags this week

OKOrhun Küpeliorhunkupeli.hashnode.dev2d ago · 5 min read

The Numbers: Benchmarking My LLM Gateway on a H100

A couple of weeks ago I wrote about rewriting my LLM gateway to bring it from MVP to production. The architectural claims were; multi-tenancy, hybrid inference , sub-5ms overhead. So I benchmarked it

0

ARAman Rajblogs.amanraj.me6d ago · 7 min read

Deploying Qwen3 235B on JarvisLabs with vLLM

This guide explains how to deploy Qwen3 235B on JarvisLabs and expose it as an OpenAI-compatible API endpoint. We will deploy this model: Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 We will serve it with

0

VGVinayak Gajarevinayakgajare.hashnode.devJun 10 · 8 min read

How I Instrumented vLLM on Kubernetes: The Dashboards, Queries, and SLOs

A practical observability setup for LLM inference on KServe — and the one-line misconfiguration it caught. LLM serving breaks the assumptions behind ordinary service dashboards. A single "request late

0

KEKarim El Jamalijamali.hashnode.devMay 17 · 10 min read

Gateway API Inference Extensions: An AI Aware Load Balancer

Introduction If you’ve been anywhere near the Kubernetes ecosystem lately, you’ve probably heard the buzz around Gateway API — and the ongoing “Ingress is going away” narrative. I’ve covered Gateway A

0

SLShaun Liewshaunliew.hashnode.devMay 16 · 14 min read

Running Qwen3-VL on DGX Spark: Transformers vs vLLM vs SGLang

I've been running Qwen3-VL locally for a while now, mostly with the standard from_pretrained() setup. It works, but it's slow. So, I kept wondering whether switching to vLLM or SGLang would actually m

0

OKOrhun Küpeliorhunkupeli.hashnode.devMay 14 · 5 min read

MVP to Mission-Critical: The Idea Behind My LLM Gateway Rewrite

Several months ago I decided to play around with my first LLM gateway prototype which I simply used LiteLLM with some benefits on top. Then I did the math to find out how far it was from the productio

0

JKJangwook Kimeffloow.hashnode.devMay 9 · 10 min read

SpecKV: Adaptive Speculative Decoding with Dynamic Gamma

Every production LLM deployment using speculative decoding is likely running a fixed speculation length of γ=4. That number comes from early benchmarks, it has been copy-pasted across blog posts and framework defaults, and almost nobody questions it....

0

강강문규devsnack.hashnode.devMay 6 · 10 min read

Qwen3.6 on DGX Spark: vLLM + NVFP4 + DFlash vs llama.cpp — 2x Faster at 88–104 tok/s

TL;DR — I was happily running Qwen3.6 on llama.cpp. Then I saw claims of 2× speed with vLLM + NVFP4 + DFlash. So I installed it, fought through crashes, and measured it myself. Verdict: it's real. 88–

0

AWAlan Westalan-west.hashnode.devMay 3 · 5 min read

Running LLMs on Windows: Native vLLM vs WSL vs llama.cpp Compared

The Windows local LLM story just got interesting. Someone recently demonstrated Qwen3's 27B model running at 72 tokens per second on an RTX 3090 — natively on Windows. No WSL. No Docker. Just a portable vLLM launcher. If you've been running local mod...

0

JKJangwook Kimeffloow.hashnode.devApr 28 · 11 min read

vLLM 0.8: Native Llama 4 MoE Routing Explained

Mixture-of-Experts models have dominated the open-weight frontier in 2026. Llama 4 Scout (17B-16E), Llama 4 Maverick (17B-128E), DeepSeek V4-Pro (1.6T-49B active), and Qwen3.6-Plus all use sparse expert routing to scale parameters without proportiona...

0

#vllm

Search Hashnode

#vllm

Explore Hashnode

Trending tags this week

The Numbers: Benchmarking My LLM Gateway on a H100

Deploying Qwen3 235B on JarvisLabs with vLLM

How I Instrumented vLLM on Kubernetes: The Dashboards, Queries, and SLOs

Gateway API Inference Extensions: An AI Aware Load Balancer

Running Qwen3-VL on DGX Spark: Transformers vs vLLM vs SGLang

MVP to Mission-Critical: The Idea Behind My LLM Gateway Rewrite

SpecKV: Adaptive Speculative Decoding with Dynamic Gamma

Qwen3.6 on DGX Spark: vLLM + NVFP4 + DFlash vs llama.cpp — 2x Faster at 88–104 tok/s

Running LLMs on Windows: Native vLLM vs WSL vs llama.cpp Compared

vLLM 0.8: Native Llama 4 MoE Routing Explained