Tag feed

#llm-inference

8 posts0 followers

Explore Hashnode

Alternatives

BKBrian Kingsolodev.app4d ago · 12 min read

Installing vLLM onto EVO-X2.

Abstract. I download a vLLM Docker image that supports AMD, and the ROCm GPU driver that runs on the Strix Halo platform (AI Max+ 395 APU). I connect the running Docker container to local GGUF AI mode

0

JKJangwook Kimeffloow.hashnode.devMay 9 · 7 min read

Adaptive KV-Cache Quantization: How 'Don't Waste Bits' Cuts On-Device LLM Latency by 17%

Running LLMs on-device means fighting two constraints simultaneously: memory and latency. The KV-cache — the buffer that stores past token representations so the model does not recompute them — is often the bottleneck on both fronts. A paper publishe...

0

JKJangwook Kimeffloow.hashnode.devMay 9 · 10 min read

SpecKV: Adaptive Speculative Decoding with Dynamic Gamma

Every production LLM deployment using speculative decoding is likely running a fixed speculation length of γ=4. That number comes from early benchmarks, it has been copy-pasted across blog posts and framework defaults, and almost nobody questions it....

0

Eeyanpenai-is-coming.hashnode.devMay 3 · 6 min read

Multiple Independent Questions: Batch Into One Request or Split Into Many? — An Analysis of LLM Concurrent Processing

When you have 5 unrelated questions, should you pack them into one message to the LLM, or send 5 requests simultaneously? Which is faster? The Short Answer Splitting into multiple independent parallel requests is almost always faster. This isn't a g...

0

AKAnup Karanjkarwowhow.hashnode.devMay 2 · 10 min read

Google TurboQuant: 6x KV Cache Compression Changes AI Inference Economics

The key-value cache is the most expensive part of running a large language model — and until now, nobody had solved it without sacrificing accuracy. At ICLR 2026, Google Research published TurboQuant: a two-stage compression algorithm that reduces KV...

0

GGowthamyobitel.hashnode.devApr 20 · 5 min read

I Tested GPT-OSS-20B Against the Field. The Architecture Explains Everything

Most posts about GPT-OSS-20B lead with the headline benchmark and stop there. This one goes a level deeper, because once you understand the architecture, the results stop being surprising and start be

0

MVMohit Vermaaiwithmohit.hashnode.devMar 25 · 2 min read

We Cut LLM Inference Carbon Emissions by 35% Using SEAL Framework

We Cut LLM Inference Carbon Emissions by 35% Using SEAL Framework LLM inference workloads double every 6–9 months. Most teams track latency & cost-per-token. Almost nobody tracks carbon emissions per request. We cut ours by 35% using the SEAL framewo...

0

VAVijayakumar Arumuga Nadarblog.neevcloud.comJan 21 · 5 min read

Low-Latency LLM Inference on Multi-GPU Cloud Systems

TL;DR Low-latency LLM inference is now a business-critical capability, not a research luxury, especially for real-time AI products in India’s fast-scaling digital economy. Multi-GPU LLM inference on cloud GPUs is the only viable path to sustain per...

0

#llm-inference

Search Hashnode

#llm-inference

Explore Hashnode

Installing vLLM onto EVO-X2.

Adaptive KV-Cache Quantization: How 'Don't Waste Bits' Cuts On-Device LLM Latency by 17%

SpecKV: Adaptive Speculative Decoding with Dynamic Gamma

Multiple Independent Questions: Batch Into One Request or Split Into Many? — An Analysis of LLM Concurrent Processing

Google TurboQuant: 6x KV Cache Compression Changes AI Inference Economics

I Tested GPT-OSS-20B Against the Field. The Architecture Explains Everything

We Cut LLM Inference Carbon Emissions by 35% Using SEAL Framework

Low-Latency LLM Inference on Multi-GPU Cloud Systems

Trending tags this week