Excellent points, especially the distinction between request rate and in-flight concurrency. A token bucket controls how quickly requests arrive, but it doesn’t guarantee that the backend can handle the number of requests still running when latency increases.

For LLM workloads, combining rate limits with a concurrency cap, reserving an upper bound on token usage and reconciling it against actual usage makes the protection much more robust. The client disconnect scenario is equally important because cancellation doesn’t necessarily stop the provider from processing or billing for the request.

Testing slowdowns and client disconnects separately is a great addition to the failure-testing checklist. Thanks for sharing these practical considerations!