The 1,068 tokens/second claim needs a latency contract alongside the output count. Parallel refinement can make a completed sequence arrive quickly without giving an interactive client an equally early, stable prefix to display.
I would compare time to first committed text, time to final usable output and the refinement budget at several requested output lengths. Include constrained JSON and infilling separately, since a fluent draft is not the same stopping condition as a valid completed edit. That would test the architecture's application advantage without assuming an OpenAI-compatible response format also implies identical streaming behavior.
Ahmet Özel
AI Engineer. Computer Vision, RAG and LLM agents.
The 1,068 tokens/second claim needs a latency contract alongside the output count. Parallel refinement can make a completed sequence arrive quickly without giving an interactive client an equally early, stable prefix to display.
I would compare time to first committed text, time to final usable output and the refinement budget at several requested output lengths. Include constrained JSON and infilling separately, since a fluent draft is not the same stopping condition as a valid completed edit. That would test the architecture's application advantage without assuming an OpenAI-compatible response format also implies identical streaming behavior.