Discussion on "Why AI Applications Need Background Workflows"

Aansh Malhotra · 2026-09-07T17:21:34.897Z

AI applications often start with a simple idea: a user sends a request, the application calls an AI model, and a response comes back. That model works well when the task takes a few seconds. But produ

Per-step retries are the right unit, and they pull a second requirement in behind them: each step has to leave a record of what it produced and from which input, or a retry can silently redo work against a newer version of the document than the step before it saw. We run intake this way for a house — assessor harvest, facade vision, sketch reconciliation, compression — and the thing that made retries safe wasn't the queue, it was giving every step's output a source and a version, so a re-run that disagrees with the earlier run shows up as a conflict instead of overwriting it.

That’s a really good point. Retries are only truly safe when each step knows exactly what input it worked on and what output it produced. Versioning those outputs also makes conflicts visible instead of letting a retry overwrite newer data. Definitely something worth considering for production workflows.

The argument gets stronger if you add why the request-response version fails badly rather than gracefully. A long AI call inside an HTTP request does not just make the user wait - it holds memory for the whole duration, so under load you do not get a slow service, you get an OOM kill. Every in-flight request is sitting on a context, and the failure arrives all at once. Two things worth pairing with the workflow queue. Idempotency keys, because retries are the entire point of a durable workflow and an agent step that charges something or sends an email twice is worse than one that failed. And a bound on queue depth with early rejection, since accepting work you cannot finish turns a capacity problem into an outage - a fast 503 is friendlier than a job that sits for twenty minutes and times out. Separating queue wait from execution time in your metrics is the other habit that pays off: they demand opposite fixes, and a single p95 blends them into something you cannot act on.

Absolutely agree. The memory/scale point is something I should have emphasized more a long-running request can become a real problem once multiple requests are running at the same time. Idempotency and queue limits are also great points, especially for workflows that can trigger side effects. And separating queue time from execution time in metrics makes a lot of sense for knowing where the actual bottleneck is.

Thanks for adding these points!

One concrete version of the memory point, if you ever expand it: the number to watch is concurrent in-flight requests multiplied by average context size, not request rate. Rate looks healthy right up until it does not, because duration is what multiplies it. Moving the work to a queue is what converts that unbounded product into a worker count you actually chose.

Search Hashnode

Why AI Applications Need Background Workflows

Responses(5)