Tag feed

#quantization

50 posts0 followers

Explore Hashnode

Alternatives

Trending tags this week

ISilya sergeevkrauncher.hashnode.devAug 19 · 12 min read

Qwen3.8 FP8: does the cheap card really save?

Cost of the Answer — issue #2. Issue #1 closed on a promise. The FP8 checkpoint fits a 32 GB consumer card that BF16 cannot rent, and the consumer card costs about half as much per hour as the 96 GB w

0

ISilya sergeevkrauncher.hashnode.devAug 17 · 9 min read

Qwen3.8: is FP8 worth it under transformers?

Cost of the Answer — issue #1. Qwen3.8-27B ships in two precisions on the HuggingFace hub. The FP8 checkpoint is roughly half the byte count of the BF16 one, and I wanted to know what that halving buy

0

JTJoTeq the Firstconfamnode.hashnode.devJul 14 · 6 min read

Quantizing MedGemma to INT4 (GPTQ/W4A16): Everything That Broke Along the Way

Quantized Google's MedGemma-1.5-4B (a medical vision-language model) to INT4 (W4A16) via llm-compressor's GPTQModifier, for self-hosted deployment. 8.6 GB in BF16 -> 5.2 GB quantized. Full step-by-ste

0

WSwesley schollkonjo.hashnode.devJul 13 · 21 min read

I Couldn't Find a Local LLM Tool Fast Enough, So I Built My Own Called Squish

This requires Apple Silicon (M-series) and macOS 13 (Ventura) or later. If you're on Intel, Linux, or Windows, the numbers in this article will not apply to your hardware. TL;DR: Squish is a local LLM

0

AAAbstract Algorithmsabstractalgorithms.hashnode.devJul 2 · 1 min read

Quantization: GPTQ vs AWQ

Quantization reduces the memory footprint of an LLM by converting weight numerical values from 16-bit floating points (FP16) to 4-bit or 8-bit integers. This allows massive models to run on affordable

0

MGManav Guptamanavgupta.hashnode.devJun 20 · 8 min read

How quantization keeps vector search in RAM

I've been building a RAG system over a terabyte of scanned engineering manuals tens of millions of pages, chunked into roughly six million searchable pieces. At that scale the interesting problems sto

0

AKAlexander Kerchumblog.kerchum.devJun 5 · 12 min read

Lessons From the Bottom of the Stack: Shipping a Quant

The SCLP compression algorithm — palette the exponents, sidecar the outliers, pack the rest — was a week or so of prototyping. The two posts before this one covered it end to end. This post is about t

0

AKAlexander Kerchumblog.kerchum.devJun 3 · 9 min read

From 8 Bits to 4: Sidecar, MoE, and the imatrix Trick That Worked

Last time we cut BF16 weights in half by treating the exponent as a 16-entry palette instead of an 8-bit field. SCLP8: 7.9 GB instead of 15.0, perplexity slightly better than the original, token gener

0

AKAlexander Kerchumblog.kerchum.devMay 29 · 9 min read

LLMs Use Just 16 of 256 Exponents — So We Compressed the Rest Away

Most people compressing LLM weights are fighting the same war: squeeze 7 billion floats into less memory without wrecking the model. The standard weapons are quantization schemes — map each float to a

0

SSapotaCorpsapotacorpvn.hashnode.devMay 24 · 7 min read

The 32x vector DB cost cut most teams do not know about

A B2B SaaS founder forwarded us his latest vector database invoice with one line: "this needs to come down by 80% before the next board update." The corpus had grown 3x in six months, the bill had gro

0

#quantization

Search Hashnode

#quantization

Explore Hashnode

Trending tags this week

Qwen3.8 FP8: does the cheap card really save?

Qwen3.8: is FP8 worth it under transformers?

Quantizing MedGemma to INT4 (GPTQ/W4A16): Everything That Broke Along the Way

I Couldn't Find a Local LLM Tool Fast Enough, So I Built My Own Called Squish

Quantization: GPTQ vs AWQ

How quantization keeps vector search in RAM

Lessons From the Bottom of the Stack: Shipping a Quant

From 8 Bits to 4: Sidecar, MoE, and the imatrix Trick That Worked

LLMs Use Just 16 of 256 Exponents — So We Compressed the Rest Away

The 32x vector DB cost cut most teams do not know about