One Generation for the Whole Burst: Coalescing Identical LLM Requests
The expensive moment in a cached system is not the hit. It is the first second of a burst, when two hundred copies of the same request arrive at once, all miss, and all run the model in parallel — pro
