In my last post I showed that block-quantized FP8 models on Ada GPUs (L4, L40S, RTX 4090) don't use the FP8 tensor cores in vLLM. vLLM falls back to Marlin, which does its math in 16-bit. I ended that

Markdown supported · ⌘↵ to post
No responses yet.