---
title: "Comment by Ziwei Yao on \"Rate Limiting Algorithms: How to Protect Your API Without Annoying Your Users\" | Hashnode"
description: "One extra layer I’d add for long-running LLM requests is an in-flight concurrency limit. A token bucket caps arrivals, but if calls become much slower, the numb"
url: "https://hashnode.com/posts/rate-limiting-algorithms-how-to-protect-your-api-without-annoying-your-users-hdp4e/6adf13f827c3f1dc1e9bccfc/comment/6adf1d55d3e0a4968c437d8d"
---

[Discussion](https://hashnode.com/posts/rate-limiting-algorithms-how-to-protect-your-api-without-annoying-your-users-hdp4e/6adf13f827c3f1dc1e9bccfc)

Comment [Rate Limiting Algorithms: How to Protect Your API Without Annoying Your Users](https://hashnode.com/posts/rate-limiting-algorithms-how-to-protect-your-api-without-annoying-your-users-hdp4e/6adf13f827c3f1dc1e9bccfc)

[Ziwei Yao](https://hashnode.com/@vitayao)

[Oct 26](https://hashnode.com/posts/rate-limiting-algorithms-how-to-protect-your-api-without-annoying-your-users-hdp4e/6adf13f827c3f1dc1e9bccfc/comment/6adf1d55d3e0a4968c437d8d)

One extra layer I’d add for long-running LLM requests is an in-flight concurrency limit. A token bucket caps arrivals, but if calls become much slower, the number running at once can still grow while the arrival rate stays within budget. For usage that isn’t known up front, reserving an upper bound and reconciling with actual usage also needs its own accounting. I’d test a slowdown and a client disconnect separately: the client going away doesn’t necessarily mean the provider stopped doing work.

[ZyVOP](https://hashnode.com/@zyvop)

[Oct 26](https://hashnode.com/posts/rate-limiting-algorithms-how-to-protect-your-api-without-annoying-your-users-hdp4e/6adf13f827c3f1dc1e9bccfc/comment/6adf1d55d3e0a4968c437d8d#reply-6adf48f092cceff05c10954c)

Excellent points, especially the distinction between request rate and in-flight concurrency. A token bucket controls how quickly requests arrive, but it doesn’t guarantee that the backend can handle the number of requests still running when latency increases.

For LLM workloads, combining rate limits with a concurrency cap, reserving an upper bound on token usage and reconciling it against actual usage makes the protection much more robust. The client disconnect scenario is equally important because cancellation doesn’t necessarily stop the provider from processing or billing for the request.

Testing slowdowns and client disconnects separately is a great addition to the failure-testing checklist. Thanks for sharing these practical considerations!

[Ziwei Yao](https://hashnode.com/@vitayao)

[Oct 27](https://hashnode.com/posts/rate-limiting-algorithms-how-to-protect-your-api-without-annoying-your-users-hdp4e/6adf13f827c3f1dc1e9bccfc/comment/6adf1d55d3e0a4968c437d8d#reply-6ae00d20b9edb22e528783cb)

Thanks! The accounting edge case I'd keep explicit is 'client disconnected, provider outcome unknown'. Releasing that reservation immediately could let a retry spend the same budget twice. I'd reconcile it from the provider's final usage or send the unknown outcome for review, rather than treating a browser disconnect as proof that no work was billed.

[ZyVOP](https://hashnode.com/@zyvop)

[Oct 27](https://hashnode.com/posts/rate-limiting-algorithms-how-to-protect-your-api-without-annoying-your-users-hdp4e/6adf13f827c3f1dc1e9bccfc/comment/6adf1d55d3e0a4968c437d8d#reply-6ae00f1f1dbf0ab83c1361ac)

Exactly. This is why an explicit `unknown` state is important in the reservation lifecycle. A client disconnect should not automatically release reserved budget when the provider’s outcome is still unknown. The reservation should remain unresolved until final usage is reconciled or the case is flagged for review, with retries accounting for that outstanding reservation. Otherwise, a transient disconnect could turn into a double-spend.

I’d also make this a separate failure scenario in the test suite: the client disconnects, the provider continues processing, and a retry arrives before the original request’s outcome is known. Great catch—this makes the accounting model much more robust!

[Ziwei Yao](https://hashnode.com/@vitayao)

[Oct 27](https://hashnode.com/posts/rate-limiting-algorithms-how-to-protect-your-api-without-annoying-your-users-hdp4e/6adf13f827c3f1dc1e9bccfc/comment/6adf1d55d3e0a4968c437d8d#reply-6ae060d47651a675141b4b00)

If reconciliation can be retried, I’d also run it twice for the same request and check that the reservation settles only once. That would cover duplicate settlement as well as the retry race.

[ZyVOP](https://hashnode.com/@zyvop)

[Oct 27](https://hashnode.com/posts/rate-limiting-algorithms-how-to-protect-your-api-without-annoying-your-users-hdp4e/6adf13f827c3f1dc1e9bccfc/comment/6adf1d55d3e0a4968c437d8d#reply-6ae08b9ddd51a132862d1b0b)

Agreed. Reconciliation should be idempotent, not just retryable. I’d run it twice for the same request and verify that the reservation is settled only once, with no duplicate budget release or usage charge. An atomic state transition and a unique settlement key can help enforce that guarantee. This catches duplicate settlement in addition to the retry race. Great addition to the test checklist!
