Putting latency SLA and context length in as first-class inputs is the right call, since those drive KV-cache memory and effective batch size more than parameter count does. The output I'd stress-test is the batching config, because the p50-versus-p99 gap under real burst traffic is where sizing usually breaks. Does it model prefill and decode separately, or assume steady-state throughput?
Kartik N V J K
AI Developer | Making AI reliable, trustworthy & accessible to everyone | Active community contributor
Putting latency SLA and context length in as first-class inputs is the right call, since those drive KV-cache memory and effective batch size more than parameter count does. The output I'd stress-test is the batching config, because the p50-versus-p99 gap under real burst traffic is where sizing usually breaks. Does it model prefill and decode separately, or assume steady-state throughput?