The Developer Publishing Hub
One platform to write, cross-post to Dev.to, Medium, WordPress, Hashnode, and Bluesky, and protect your SEO with canonical links. Zero paywalls and full content ownership.
Markdown supported · ⌘↵ to post
For variable request costs, I would distinguish an empty bucket from a request that costs more than the bucket can ever hold. Waiting and retrying can fix the first case but not the second, so the client needs a different explanation
One extra layer I’d add for long-running LLM requests is an in-flight concurrency limit. A token bucket caps arrivals, but if calls become much slower, the number running at once can still grow while the arrival rate stays within budget. For usage that isn’t known up front, reserving an upper bound and reconciling with actual usage also needs its own accounting. I’d test a slowdown and a client disconnect separately: the client going away doesn’t necessarily mean the provider stopped doing work.
melonape
One more thing for the IP section: on IPv6, key on the /64, not the full address. One machine usually gets a whole /64 and can rotate through addresses for free, so per-address limits barely slow it down.
Mobile carriers are the opposite problem. A lot of people sit behind carrier-grade NAT, so in some regions one IPv4 can be a whole neighborhood. Per-IP limits only really make sense as a loose outer layer.