Modeling latency as max of compute time and memory time is the framing that finally makes the memory-bound versus compute-bound crossover intuitive. At small batch sizes you're paying for weight loads that serve almost nobody, which is why throughput jumps so much as you fill the batch. Where did the crossover land for the model you tested, and how much did KV cache growth pull it back down?
Kartik N V J K
AI Developer | Making AI reliable, trustworthy & accessible to everyone | Active community contributor
Modeling latency as max of compute time and memory time is the framing that finally makes the memory-bound versus compute-bound crossover intuitive. At small batch sizes you're paying for weight loads that serve almost nobody, which is why throughput jumps so much as you fill the batch. Where did the crossover land for the model you tested, and how much did KV cache growth pull it back down?