This is exactly where “acknowledge” and “process” should become two different transactions. We return the provider’s required 200/ACK after validating the signature and persisting a minimal immutable envelope, then let a queue handle the 15 calls. The envelope gets an idempotency key, correlation ID, attempt count, and a terminal status; each downstream side effect checks that key before writing.
The two failure modes worth designing up front are: the queue is unavailable when the webhook arrives, and the worker succeeds but crashes before recording success. A durable inbox plus idempotent workers covers both better than simply moving the awaits to Promise.all.
We ran into the same timeout/retry behavior while building Atlantic’s voice and CRM pipelines. The durable pattern was fast ACK + durable inbox + replayable worker + DLQ, with the final CRM write carrying the original event evidence.
I’m Ege, co-founder of Atlantic, sharing our implementation experience.
This is exactly where “acknowledge” and “process” should become two different transactions. We return the provider’s required 200/ACK after validating the signature and persisting a minimal immutable envelope, then let a queue handle the 15 calls. The envelope gets an idempotency key, correlation ID, attempt count, and a terminal status; each downstream side effect checks that key before writing.
The two failure modes worth designing up front are: the queue is unavailable when the webhook arrives, and the worker succeeds but crashes before recording success. A durable inbox plus idempotent workers covers both better than simply moving the awaits to Promise.all.
We ran into the same timeout/retry behavior while building Atlantic’s voice and CRM pipelines. The durable pattern was fast ACK + durable inbox + replayable worker + DLQ, with the final CRM write carrying the original event evidence.
I’m Ege, co-founder of Atlantic, sharing our implementation experience.