Claude API in Production: Rate Limits, Retries, and Batching
The first real incident with an LLM API is almost never about answer quality. It's a wall of 429 errors at 9:00 on a Monday, when every scheduled job in the company wakes up at once. This post covers
Responses
Markdown supported · ⌘↵ to post
One thing I’d watch in a document-processing pipeline is what happens when a batch partially fails while the retry queue is already backing up.
Say you're processing millions of support events, and each result updates a customer record plus emits a downstream event. If a worker crashes after saving the record but before acknowledging the batch item, replaying it can duplicate the downstream event even if the database write itself is idempotent. Meanwhile, retrying every failed item immediately can push the API straight back into the same rate limit.
I’d track each item by a stable ID, persist its processing state, and use an outbox or equivalent mechanism for downstream side effects. Then retries can target only failed items, with concurrency adjusted against token consumption rather than request count alone.
The part I find tricky is deciding when to requeue an item versus parking it for investigation. How do you handle that boundary when the batch contains a mix of transient API failures and failures in your own processing pipeline?