Our LLM service had no backpressure. The provider got 7x slower and our p99 got 25x worse.
TL;DR. Our summarization endpoint holds a p99 under 3 seconds. On 21/05 our provider degraded: median call time went from 620ms to about 4.3 seconds, roughly 7x. Our p99 went to 35 seconds, roughly 25
