The 85.9 ms handoff example makes output-length prediction a consequential admission decision: a 128-token completion and a 512-token completion repay the same transfer very differently, but the scheduler does not know the final length yet.
I would evaluate routing regret by predicted-length bucket, including requests that end early, rather than reporting only decisions made with the observed final length. The replay can compare the selected path with the alternative under the same arrival trace and state placement. That would distinguish an attractive offline break-even calculation from a routing policy that still helps when its output-length estimate is wrong.
The 85.9 ms handoff example makes output-length prediction a consequential admission decision: a 128-token completion and a 512-token completion repay the same transfer very differently, but the scheduler does not know the final length yet.
I would evaluate routing regret by predicted-length bucket, including requests that end early, rather than reporting only decisions made with the observed final length. The replay can compare the selected path with the alternative under the same arrival trace and state placement. That would distinguish an attractive offline break-even calculation from a routing policy that still helps when its output-length estimate is wrong.