Tag feed

#quantization

46 posts0 followers

Explore Hashnode

Alternatives

Trending tags this week

MGManav Guptamanavgupta.hashnode.devJun 20 · 8 min read

How quantization keeps vector search in RAM

I've been building a RAG system over a terabyte of scanned engineering manuals tens of millions of pages, chunked into roughly six million searchable pieces. At that scale the interesting problems sto

0

AKAlexander Kerchumblog.kerchum.devJun 5 · 12 min read

Lessons From the Bottom of the Stack: Shipping a Quant

The SCLP compression algorithm — palette the exponents, sidecar the outliers, pack the rest — was a week or so of prototyping. The two posts before this one covered it end to end. This post is about t

0

AKAlexander Kerchumblog.kerchum.devJun 3 · 9 min read

From 8 Bits to 4: Sidecar, MoE, and the imatrix Trick That Worked

Last time we cut BF16 weights in half by treating the exponent as a 16-entry palette instead of an 8-bit field. SCLP8: 7.9 GB instead of 15.0, perplexity slightly better than the original, token gener

0

AKAlexander Kerchumblog.kerchum.devMay 29 · 9 min read

LLMs Use Just 16 of 256 Exponents — So We Compressed the Rest Away

Most people compressing LLM weights are fighting the same war: squeeze 7 billion floats into less memory without wrecking the model. The standard weapons are quantization schemes — map each float to a

0

EECaricatozerotomodel.hashnode.devMay 16 · 13 min read

Turbovec: A Practical Manual for Training-Free Vector Quantization in Rust and Python

A hands on guide for AI engineers who care about embedding compression, retrieval latency, and not having to rebuild their index every time the data shifts. If you build retrieval augmented generatio

0

JKJangwook Kimeffloow.hashnode.devMay 9 · 7 min read

Adaptive KV-Cache Quantization: How 'Don't Waste Bits' Cuts On-Device LLM Latency by 17%

Running LLMs on-device means fighting two constraints simultaneously: memory and latency. The KV-cache — the buffer that stores past token representations so the model does not recompute them — is often the bottleneck on both fronts. A paper publishe...

0

VBVlad Butacuomniforge.onlineMay 4 · 9 min read

Green AI at the Edge: When Local LLMs Save Electricity and Water

AI sustainability keeps getting framed around the wrong question: does a local LLM use less electricity and water than ChatGPT or Claude? The sharper question is: how many frontier-model calls are we

0

OWOliver Wengedge-insights.hashnode.devApr 27 · 7 min read

🚀 Edge AI & Efficient Computing | Bi-Weekly News: # 3

🎉Welcome to the 3rd post of Edge AI & Efficient Computing! 👋 I’m Yuqin (Oliver) Weng, and I’m incredibly passionate about pushing the boundaries of AI on constrained devices. 🧠I created this space

0

PMPROTIK MONDALnila.mndl.eu.orgApr 26 · 5 min read

The April 2026 Private AI Revolution: Running DeepSeek-R2-7B at 60 tokens/sec on Your Gaming Rig

The April 2026 Private AI Revolution: Running DeepSeek-R2-7B at 60 tokens/sec on Your Gaming Rig Real numbers: I'm getting 61.4 tokens per second on a consumer RTX 4080 with DeepSeek-R2-7B. Cost? $0.00/month vs $240/month cloud inference. Zero depend...

0

OMO Melinggothartech.hashnode.devApr 26 · 18 min read

Edge AI Inference: Running Models at the CDN Layer

Originally published at Gothar Tech Part of our 2025 software architecture series. Edge AI Inference: Running Models at the CDN Layer The fastest inference call is the one that never crosses an ocean. For two decades, CDNs existed to cache bytes: i...

0

#quantization

Search Hashnode

#quantization

Explore Hashnode

Trending tags this week

How quantization keeps vector search in RAM

Lessons From the Bottom of the Stack: Shipping a Quant

From 8 Bits to 4: Sidecar, MoE, and the imatrix Trick That Worked

LLMs Use Just 16 of 256 Exponents — So We Compressed the Rest Away

Turbovec: A Practical Manual for Training-Free Vector Quantization in Rust and Python

Adaptive KV-Cache Quantization: How 'Don't Waste Bits' Cuts On-Device LLM Latency by 17%

Green AI at the Edge: When Local LLMs Save Electricity and Water

🚀 Edge AI & Efficient Computing | Bi-Weekly News: # 3

The April 2026 Private AI Revolution: Running DeepSeek-R2-7B at 60 tokens/sec on Your Gaming Rig

Edge AI Inference: Running Models at the CDN Layer