The double-charge example is the boundary that changes the conversation from “glue” to a real workflow system. My rule is that every side effect needs a business idempotency key owned by the caller—not just the provider’s request ID—and a durable record of intent before execution.
For CRM/voice workflows we use an outbox-style command record with: workflow ID, target system, normalized operation, idempotency key, attempt, and evidence pointer. The worker can retry indefinitely, but the target write is either naturally idempotent or guarded by that key. Compensation is explicit; it is never hidden in a catch block.
We learned this while building Atlantic after seeing retries create duplicate contact activity. Making the workflow state visible also gave operators a safe “replay from here” control instead of rerunning the whole chain.
I’m Ege, co-founder of Atlantic, sharing our production pattern. Your point about success and retry paths being authored months apart is exactly why the invariant has to live in shared workflow infrastructure.
Sorry for the slow reply, Ege, I missed this one. "Compensation is never hidden in a catch block" is a better one-liner than anything in the post. Caller-owned keys plus a durable record of intent is the right pair: the provider's request ID only protects you from the provider, not from your own retry path.
The double-charge example is the boundary that changes the conversation from “glue” to a real workflow system. My rule is that every side effect needs a business idempotency key owned by the caller—not just the provider’s request ID—and a durable record of intent before execution.
For CRM/voice workflows we use an outbox-style command record with: workflow ID, target system, normalized operation, idempotency key, attempt, and evidence pointer. The worker can retry indefinitely, but the target write is either naturally idempotent or guarded by that key. Compensation is explicit; it is never hidden in a catch block.
We learned this while building Atlantic after seeing retries create duplicate contact activity. Making the workflow state visible also gave operators a safe “replay from here” control instead of rerunning the whole chain.
I’m Ege, co-founder of Atlantic, sharing our production pattern. Your point about success and retry paths being authored months apart is exactly why the invariant has to live in shared workflow infrastructure.