Lessons From the Bottom of the Stack: Shipping a QuantShipping a 4-Bit LLM Quant into llama.cpp12 min read·blog.kerchum.devJun 5
From 8 Bits to 4: Sidecar, MoE, and the imatrix Trick That WorkedPushing SCLP exponent-palette compression to 4 bits/weight: per-block palettes, a fused MoE GEMV, and an imatrix sidecar that beats Q4_K.9 min read·blog.kerchum.devJun 3
LLMs Use Just 16 of 256 Exponents — So We Compressed the Rest Away2× compression on Llama-3-8B — and perplexity went down.9 min read·blog.kerchum.dev2May 29
How to move a directory from one git repo to another (or new) without losing historyMake copy of repo git clone dirtySourceRepo newSourceRepo OR clone from actual git repo and prevent push git remote set-url --push origin no_push Make sure to checkout the correct branch before the next step. Cloning from another local directory al...1 min read·blog.kerchum.devNov 24, 2021