Breaking the inference wall into where latency and cost actually accrue is the useful move, since most teams treat it as one number when prefill and decode have completely different cost curves. The lever that surprised me most was batching, where throughput and per-request latency pull against each other so tuning one silently taxes the other. Since this is part one of four, are you getting into KV-cache pressure, because that's where my cost math usually falls apart?
Kartik N V J K
Breaking the inference wall into where latency and cost actually accrue is the useful move, since most teams treat it as one number when prefill and decode have completely different cost curves. The lever that surprised me most was batching, where throughput and per-request latency pull against each other so tuning one silently taxes the other. Since this is part one of four, are you getting into KV-cache pressure, because that's where my cost math usually falls apart?