---
title: "Discussion on \"I Kill My Workers on Purpose\" | Hashnode"
description: "Discussion on \"I Kill My Workers on Purpose\". Building a crash-safe workflow engine on PostgreSQL: one transaction per decision, leases instead of locks, and a test that kills its own worker"
url: "https://hashnode.com/posts/crash-safe-workflow-engine-postgres/6afcf385577ca6d1f890dc51"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [I Kill My Workers on Purpose](https://blog.jdcollins.io/crash-safe-workflow-engine-postgres?utm_source=hashnode&utm_medium=feed)

Building a crash-safe workflow engine on PostgreSQL: one transaction per decision, leases instead of locks, and a test that kills its own worker

18h ago · 10 min read · blog.jdcollins.io  
Read the article

-   [#postgresql](https://hashnode.com/tag/postgresql)
-   [#typescript](https://hashnode.com/tag/typescript)
-   [#distributed-systems](https://hashnode.com/tag/distributed-systems)
-   [#software-architecture](https://hashnode.com/tag/software-architecture)
-   [#testing](https://hashnode.com/tag/testing)

## Written by

[Jason Collins](https://hashnode.com/@jdcollins)

@jdcollins · Macomb, MI

  
[View profile](https://hashnode.com/@jdcollins)

Analytics and AI leader building governed AI orchestration in public.

I lead analytics and a growing portfolio of enterprise AI work, where I’ve learned that the hard part of AI isn’t the model. It’s accountability: who approved what, what an agent was allowed to do, and how you prove it. I’m building Canonik in public, a neutral orchestration engine for work done by people, AI agents, and systems, and writing here about the decisions behind it.

## Responses 8

Markdown supported · ⌘↵ to post

[Mike Dabydeen](https://hashnode.com/@firelinks)

[6h ago](https://hashnode.com/posts/crash-safe-workflow-engine-postgres/6afcf385577ca6d1f890dc51/comment/6afd96c74300b22e5f05a8fd)

When the idempotency keys go in, I'd build them from the job id and step, never from the attempt number. The attempt counter is the right fence for your own tables, but if it ends up in the key, a reclaimed job hands the other system a key it has never seen, and the duplicate ticket gets created anyway. The fence has to change on every claim; the key has to stay the same across them.

The heartbeats you're planning can carry the same token. If the heartbeat runs UPDATE jobs SET lease\_expires\_at = ... WHERE id = $1 AND attempts = $2 and gets zero rows back, the worker has already lost the job and should stop before its next external call, instead of learning that when its success write matches nothing. For a system that doesn't accept idempotency keys, writing the job id into the record it creates, in an external reference field or a label, gives the retry something to search for before it creates another one.

[Jason Collins](https://hashnode.com/@jdcollins)

[5h ago](https://hashnode.com/posts/crash-safe-workflow-engine-postgres/6afcf385577ca6d1f890dc51/comment/6afd96c74300b22e5f05a8fd#reply-6afda24d908e8268e51ce1ac)

This is really helpful. Thank you!

You're right. I was mentally carrying the attempt number around because it's already part of the claim lifecycle. Thinking about the idempotency key as the identity of the work itself, rather than a particular attempt, makes the distinction much clearer.

The plan already had fenced heartbeats, but not the part where losing the lease stops the handler before its next outside call. That’s a much better moment to find out than waiting for the final write to match nothing, so I’m adding that.

I also like the external reference idea for systems that don’t support idempotency keys. Search-before-create still leaves a small window if two attempts overlap, but it turns many duplicates from “create another record” into “find the one that’s already there.”

All three are going into the plan. You rock!

[QRiQRa Team](https://hashnode.com/@qriqra)

[14h ago](https://hashnode.com/posts/crash-safe-workflow-engine-postgres/6afcf385577ca6d1f890dc51/comment/6afd2b5f54af9423416415b8)

The point that a row lock and a revision number catch different failures is easy to miss. Simultaneous writers need the lock; a decision from a screen that went stale needs the revision. Treating them as one check would leave one of those gaps open. The fencing token from the claim number, with no extra column, is a clean way to drop a late result. And the backoff on expired leases is worth keeping: an overdue worker may only be slow, and an immediate retry of a job that crashed a worker can take the next one down with it. The honest limit at the end also belongs in the same picture: the database can be exactly once, while a call to another system may still run more than once if the worker disappears after the call and before the record.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[15h ago](https://hashnode.com/posts/crash-safe-workflow-engine-postgres/6afcf385577ca6d1f890dc51/comment/6afd146254af94234163cfcd)

The SIGKILL test is the honest proof here, since a clean shutdown never shows the real failure mode. One detail I would watch is the lease length: too short and a slow worker loses a job mid-step, too long and recovery waits, so I would tie it to the slowest step plus a margin.

[Jason Collins](https://hashnode.com/@jdcollins)

[8h ago](https://hashnode.com/posts/crash-safe-workflow-engine-postgres/6afcf385577ca6d1f890dc51/comment/6afd146254af94234163cfcd#reply-6afd7b1e500f562994b30c21)

Nice catch! I'd thought about slow handlers and lease length early on...then lost sight of it.

I should be able to get around the issue by having each handler declare its expected duration and workers send heartbeats to extend the lease while a long-running job is still healthy. That lets slow jobs keep their claim without forcing every lease to be long, so recovery can still happen quickly when a worker really does die.

I’ve added it to the plan. Thanks!

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[7h ago](https://hashnode.com/posts/crash-safe-workflow-engine-postgres/6afcf385577ca6d1f890dc51/comment/6afd146254af94234163cfcd#reply-6afd926f6bb10234cd829b49)

Heartbeats plus declared durations is a clean fix, since the lease can stay short for fast recovery while slow jobs keep their claim. One edge I would still test is a stalled heartbeat on a healthy handler, so a job is not reclaimed mid-step. Keen to see how it holds up in the next SIGKILL run.

[Ascendra Ventures](https://hashnode.com/@ascendra-ventures)

[16h ago](https://hashnode.com/posts/crash-safe-workflow-engine-postgres/6afcf385577ca6d1f890dc51/comment/6afd11afce409c14792a6d00)

The SIGKILL test is a great way to prove it: ten jobs claimed, the worker gone with no cleanup, and a second worker finishing all ten with every step completed exactly once. Keeping decide() pure, so every bug lands either in the decision or in recording it, is the part I would copy first.

We worked on the same promise at Ascendra Ventures for a seed-stage talent platform. Their hiring pipeline did not fail loudly, it just stopped. We rebuilt the profile processor as a durable workflow: urgent profiles run now, the rest are batched, and polling takes over when callbacks go quiet. The goal we wrote down was that every candidate finishes or visibly fails and never gets stuck in the middle. The numbers we shared afterwards were 99.99% system reliability and $0.004 per candidate imported, but the rule was what shaped the design.

How do you plan to handle a job that keeps crashing whichever worker picks it up, so it does not take down each new worker in turn?

[Jason Collins](https://hashnode.com/@jdcollins)

[8h ago](https://hashnode.com/posts/crash-safe-workflow-engine-postgres/6afcf385577ca6d1f890dc51/comment/6afd11afce409c14792a6d00#reply-6afd829cacee3741c2c46fdc)

Thanks! That’s a good question.

Most of the protection is already there. Every claim counts as an attempt, including reclaims after a worker dies. If a job keeps crashing whichever worker picks it up, it eventually hits the attempt cap and is marked failed instead of cycling forever.

Reclaimed jobs also back off before they’re retried, so a bad job can only take down one worker at a time. It doesn’t immediately get handed to the next worker and cascade through the whole pool.

The part I still want to improve is isolation. Right now, if a handler crashes the process, it takes the rest of that worker’s batch down with it. Running handlers in their own processes would shrink that blast radius a lot. That’s on the list.

Your “finishes or visibly fails” rule made me realize I’m only solving the infrastructure side of the problem today. The job fails, but the workflow isn’t told. From the request owner’s perspective, that’s still a thing stuck in the middle.

The fix I have planned is to turn terminal job failures into workflow events. Then the workflow knows the step failed and can route it to whoever can act on it, like the request owner when the input is bad or an admin when an external system is down.
