The MTP-3 numbers are the standout here, more than doubling output
tok/s/user (130 → 307 on FP8, 133 → 304 on NVFP4) for basically a config
flag is a huge free win, and the fact that MTP-1 barely helps (and can
hurt at higher concurrency) is a useful warning against assuming "more
speculative tokens = always better."
The TP4 × PP3 explanation for why TP12 doesn't work (64 attention heads
not divisible by 12) is exactly the kind of constraint that isn't
obvious until you hit it, good to have that spelled out instead of
discovering it from a cryptic vLLM error.
Also appreciate that H100/A100/B200 configs were explicitly left out
rather than extrapolated from memory math, "this guide does not turn
memory estimates into deployment claims" is a good principle more
infra writeups should follow.
The MTP-3 numbers are the standout here, more than doubling output tok/s/user (130 → 307 on FP8, 133 → 304 on NVFP4) for basically a config flag is a huge free win, and the fact that MTP-1 barely helps (and can hurt at higher concurrency) is a useful warning against assuming "more speculative tokens = always better."
The TP4 × PP3 explanation for why TP12 doesn't work (64 attention heads not divisible by 12) is exactly the kind of constraint that isn't obvious until you hit it, good to have that spelled out instead of discovering it from a cryptic vLLM error.
Also appreciate that H100/A100/B200 configs were explicitly left out rather than extrapolated from memory math, "this guide does not turn memory estimates into deployment claims" is a good principle more infra writeups should follow.