Rate Limiting Algorithms: How to Protect Your API Without Annoying Your Users
Every API eventually meets a client that sends too much traffic. Sometimes it’s a buggy retry loop, sometimes a scraper, and sometimes a perfectly well-meaning customer whose cron job just got ten tim
Responses
Markdown supported · ⌘↵ to post
Ziwei Yao
Manager at Tokatlas
One extra layer I’d add for long-running LLM requests is an in-flight concurrency limit. A token bucket caps arrivals, but if calls become much slower, the number running at once can still grow while the arrival rate stays within budget. For usage that isn’t known up front, reserving an upper bound and reconciling with actual usage also needs its own accounting. I’d test a slowdown and a client disconnect separately: the client going away doesn’t necessarily mean the provider stopped doing work.