Analytics and AI leader building governed AI orchestration in public.
I lead analytics and a growing portfolio of enterprise AI work, where I’ve learned that the hard part of AI isn’t the model. It’s accountability: who approved what, what an agent was allowed to do, and how you prove it. I’m building Canonik in public, a neutral orchestration engine for work done by people, AI agents, and systems, and writing here about the decisions behind it.
Markdown supported · ⌘↵ to post
The point that a row lock and a revision number catch different failures is easy to miss. Simultaneous writers need the lock; a decision from a screen that went stale needs the revision. Treating them as one check would leave one of those gaps open. The fencing token from the claim number, with no extra column, is a clean way to drop a late result. And the backoff on expired leases is worth keeping: an overdue worker may only be slow, and an immediate retry of a job that crashed a worker can take the next one down with it. The honest limit at the end also belongs in the same picture: the database can be exactly once, while a call to another system may still run more than once if the worker disappears after the call and before the record.
The SIGKILL test is the honest proof here, since a clean shutdown never shows the real failure mode. One detail I would watch is the lease length: too short and a slow worker loses a job mid-step, too long and recovery waits, so I would tie it to the slowest step plus a margin.
The SIGKILL test is a great way to prove it: ten jobs claimed, the worker gone with no cleanup, and a second worker finishing all ten with every step completed exactly once. Keeping decide() pure, so every bug lands either in the decision or in recording it, is the part I would copy first.
We worked on the same promise at Ascendra Ventures for a seed-stage talent platform. Their hiring pipeline did not fail loudly, it just stopped. We rebuilt the profile processor as a durable workflow: urgent profiles run now, the rest are batched, and polling takes over when callbacks go quiet. The goal we wrote down was that every candidate finishes or visibly fails and never gets stuck in the middle. The numbers we shared afterwards were 99.99% system reliability and $0.004 per candidate imported, but the rule was what shaped the design.
How do you plan to handle a job that keeps crashing whichever worker picks it up, so it does not take down each new worker in turn?
Mike Dabydeen
When the idempotency keys go in, I'd build them from the job id and step, never from the attempt number. The attempt counter is the right fence for your own tables, but if it ends up in the key, a reclaimed job hands the other system a key it has never seen, and the duplicate ticket gets created anyway. The fence has to change on every claim; the key has to stay the same across them.
The heartbeats you're planning can carry the same token. If the heartbeat runs UPDATE jobs SET lease_expires_at = ... WHERE id = $1 AND attempts = $2 and gets zero rows back, the worker has already lost the job and should stop before its next external call, instead of learning that when its success write matches nothing. For a system that doesn't accept idempotency keys, writing the job id into the record it creates, in an external reference field or a label, gives the retry something to search for before it creates another one.