Comment by Ahmet Özel on "Why AI Applications Need Background Workflows"

Ahmet Özel

AI Engineer. Computer Vision, RAG and LLM agents.

The argument gets stronger if you add why the request-response version fails badly rather than gracefully. A long AI call inside an HTTP request does not just make the user wait - it holds memory for the whole duration, so under load you do not get a slow service, you get an OOM kill. Every in-flight request is sitting on a context, and the failure arrives all at once. Two things worth pairing with the workflow queue. Idempotency keys, because retries are the entire point of a durable workflow and an agent step that charges something or sends an email twice is worse than one that failed. And a bound on queue depth with early rejection, since accepting work you cannot finish turns a capacity problem into an outage - a fast 503 is friendlier than a job that sits for twenty minutes and times out. Separating queue wait from execution time in your metrics is the other habit that pays off: they demand opposite fixes, and a single p95 blends them into something you cannot act on.

Aansh Malhotra

2d ago

Absolutely agree. The memory/scale point is something I should have emphasized more a long-running request can become a real problem once multiple requests are running at the same time. Idempotency and queue limits are also great points, especially for workflows that can trigger side effects. And separating queue time from execution time in metrics makes a lot of sense for knowing where the actual bottleneck is.

Thanks for adding these points!

One concrete version of the memory point, if you ever expand it: the number to watch is concurrent in-flight requests multiplied by average context size, not request rate. Rate looks healthy right up until it does not, because duration is what multiplies it. Moving the work to a queue is what converts that unbounded product into a worker count you actually chose.

Search Hashnode