Tag feed

#quantization

48 posts0 followers

Explore Hashnode

Alternatives

Trending tags this week

JTJoTeq the Firstconfamnode.hashnode.dev6d ago · 6 min read

Quantizing MedGemma to INT4 (GPTQ/W4A16): Everything That Broke Along the Way

Quantized Google's MedGemma-1.5-4B (a medical vision-language model) to INT4 (W4A16) via llm-compressor's GPTQModifier, for self-hosted deployment. 8.6 GB in BF16 -> 5.2 GB quantized. Full step-by-ste

0

WSwesley schollkonjo.hashnode.devJul 13 · 21 min read

I Couldn't Find a Local LLM Tool Fast Enough, So I Built My Own Called Squish

This requires Apple Silicon (M-series) and macOS 13 (Ventura) or later. If you're on Intel, Linux, or Windows, the numbers in this article will not apply to your hardware. TL;DR: Squish is a local LLM

0

MGManav Guptamanavgupta.hashnode.devJun 20 · 8 min read

How quantization keeps vector search in RAM

I've been building a RAG system over a terabyte of scanned engineering manuals tens of millions of pages, chunked into roughly six million searchable pieces. At that scale the interesting problems sto

0

AKAlexander Kerchumblog.kerchum.devJun 5 · 12 min read

Lessons From the Bottom of the Stack: Shipping a Quant

The SCLP compression algorithm — palette the exponents, sidecar the outliers, pack the rest — was a week or so of prototyping. The two posts before this one covered it end to end. This post is about t

0

AKAlexander Kerchumblog.kerchum.devJun 3 · 9 min read

From 8 Bits to 4: Sidecar, MoE, and the imatrix Trick That Worked

Last time we cut BF16 weights in half by treating the exponent as a 16-entry palette instead of an 8-bit field. SCLP8: 7.9 GB instead of 15.0, perplexity slightly better than the original, token gener

0

AKAlexander Kerchumblog.kerchum.devMay 29 · 9 min read

LLMs Use Just 16 of 256 Exponents — So We Compressed the Rest Away

Most people compressing LLM weights are fighting the same war: squeeze 7 billion floats into less memory without wrecking the model. The standard weapons are quantization schemes — map each float to a

0

EECaricatozerotomodel.hashnode.devMay 16 · 13 min read

Turbovec: A Practical Manual for Training-Free Vector Quantization in Rust and Python

A hands on guide for AI engineers who care about embedding compression, retrieval latency, and not having to rebuild their index every time the data shifts. If you build retrieval augmented generatio

0

JKJangwook Kimeffloow.hashnode.devMay 9 · 7 min read

Adaptive KV-Cache Quantization: How 'Don't Waste Bits' Cuts On-Device LLM Latency by 17%

Running LLMs on-device means fighting two constraints simultaneously: memory and latency. The KV-cache — the buffer that stores past token representations so the model does not recompute them — is often the bottleneck on both fronts. A paper publishe...

0

VBVlad Butacuomniforge.onlineMay 4 · 9 min read

Green AI at the Edge: When Local LLMs Save Electricity and Water

AI sustainability keeps getting framed around the wrong question: does a local LLM use less electricity and water than ChatGPT or Claude? The sharper question is: how many frontier-model calls are we

0

OWOliver Wengedge-insights.hashnode.devApr 27 · 7 min read

🚀 Edge AI & Efficient Computing | Bi-Weekly News: # 3

🎉Welcome to the 3rd post of Edge AI & Efficient Computing! 👋 I’m Yuqin (Oliver) Weng, and I’m incredibly passionate about pushing the boundaries of AI on constrained devices. 🧠I created this space

0

#quantization

Search Hashnode

#quantization

Explore Hashnode

Trending tags this week

Quantizing MedGemma to INT4 (GPTQ/W4A16): Everything That Broke Along the Way

I Couldn't Find a Local LLM Tool Fast Enough, So I Built My Own Called Squish

How quantization keeps vector search in RAM

Lessons From the Bottom of the Stack: Shipping a Quant

From 8 Bits to 4: Sidecar, MoE, and the imatrix Trick That Worked

LLMs Use Just 16 of 256 Exponents — So We Compressed the Rest Away

Turbovec: A Practical Manual for Training-Free Vector Quantization in Rust and Python

Adaptive KV-Cache Quantization: How 'Don't Waste Bits' Cuts On-Device LLM Latency by 17%

Green AI at the Edge: When Local LLMs Save Electricity and Water

🚀 Edge AI & Efficient Computing | Bi-Weekly News: # 3