The 85.9 ms handoff example makes output-length prediction a consequential admission decision: a 128-token completion and a 512-token completion repay the same transfer very differently, but the scheduler does not know the final length yet.
I would evaluate routing regret by predicted-length bucket, including requests that end early, rather than reporting only decisions made with the observed final length. The replay can compare the selected path with the alternative under the same arrival trace and state placement. That would distinguish an attractive offline break-even calculation from a routing policy that still helps when its output-length estimate is wrong.
Thanks for reading! I like your suggestion to measure routing regret by predicted-length bucket, since it connects prediction errors to their actual decision cost. This article uses given output lengths to build intuition, while a runtime must decide before the final length is known. I’d also considered length prediction as a scheduling signal and am exploring approaches such as ELIS. Your suggestion gives me a concrete way to evaluate that next step.
That distinction between a given length and a prediction keeps the next experiment well scoped. I would include the predictor’s own latency and resource use in the selected route’s cost, especially for short completions where the handoff margin is already narrow.
A simple baseline that routes using only request features available at admission would make the comparison useful. The predicted-length policy can then earn its extra complexity by reducing realized regret, rather than winning only when its prediction work is treated as free.
Ahmet Özel
AI Engineer. Computer Vision, RAG and LLM agents.
The 85.9 ms handoff example makes output-length prediction a consequential admission decision: a 128-token completion and a 512-token completion repay the same transfer very differently, but the scheduler does not know the final length yet.
I would evaluate routing regret by predicted-length bucket, including requests that end early, rather than reporting only decisions made with the observed final length. The replay can compare the selected path with the alternative under the same arrival trace and state placement. That would distinguish an attractive offline break-even calculation from a routing policy that still helps when its output-length estimate is wrong.