---
title: "Comment by Ascendra Ventures on \"I Kill My Workers on Purpose\" | Hashnode"
description: "The SIGKILL test is a great way to prove it: ten jobs claimed, the worker gone with no cleanup, and a second worker finishing all ten with every step completed"
url: "https://hashnode.com/posts/crash-safe-workflow-engine-postgres/6afcf385577ca6d1f890dc51/comment/6afd11afce409c14792a6d00"
---

[Discussion](https://hashnode.com/posts/crash-safe-workflow-engine-postgres/6afcf385577ca6d1f890dc51)

Comment [I Kill My Workers on Purpose](https://hashnode.com/posts/crash-safe-workflow-engine-postgres/6afcf385577ca6d1f890dc51)

[Ascendra Ventures](https://hashnode.com/@ascendra-ventures)

[16h ago](https://hashnode.com/posts/crash-safe-workflow-engine-postgres/6afcf385577ca6d1f890dc51/comment/6afd11afce409c14792a6d00)

The SIGKILL test is a great way to prove it: ten jobs claimed, the worker gone with no cleanup, and a second worker finishing all ten with every step completed exactly once. Keeping decide() pure, so every bug lands either in the decision or in recording it, is the part I would copy first.

We worked on the same promise at Ascendra Ventures for a seed-stage talent platform. Their hiring pipeline did not fail loudly, it just stopped. We rebuilt the profile processor as a durable workflow: urgent profiles run now, the rest are batched, and polling takes over when callbacks go quiet. The goal we wrote down was that every candidate finishes or visibly fails and never gets stuck in the middle. The numbers we shared afterwards were 99.99% system reliability and $0.004 per candidate imported, but the rule was what shaped the design.

How do you plan to handle a job that keeps crashing whichever worker picks it up, so it does not take down each new worker in turn?

[Jason Collins](https://hashnode.com/@jdcollins)

[8h ago](https://hashnode.com/posts/crash-safe-workflow-engine-postgres/6afcf385577ca6d1f890dc51/comment/6afd11afce409c14792a6d00#reply-6afd829cacee3741c2c46fdc)

Thanks! That’s a good question.

Most of the protection is already there. Every claim counts as an attempt, including reclaims after a worker dies. If a job keeps crashing whichever worker picks it up, it eventually hits the attempt cap and is marked failed instead of cycling forever.

Reclaimed jobs also back off before they’re retried, so a bad job can only take down one worker at a time. It doesn’t immediately get handed to the next worker and cascade through the whole pool.

The part I still want to improve is isolation. Right now, if a handler crashes the process, it takes the rest of that worker’s batch down with it. Running handlers in their own processes would shrink that blast radius a lot. That’s on the list.

Your “finishes or visibly fails” rule made me realize I’m only solving the infrastructure side of the problem today. The job fails, but the workflow isn’t told. From the request owner’s perspective, that’s still a thing stuck in the middle.

The fix I have planned is to turn terminal job failures into workflow events. Then the workflow knows the step failed and can route it to whoever can act on it, like the request owner when the input is bad or an admin when an external system is down.
