The framing that each request holds GPU memory for the conversation state, not just the weights, is the part people skip when they reason about inference cost as a flat per-token number. Once the KV cache grows with sequence length, prefill and decode stop behaving like the same workload, and batching throughput bends around memory rather than FLOPs. Did Moyou's numbers separate the compute-bound prefill phase from the memory-bound decode phase, or report an aggregate tokens-per-second?