Discussion on "My Idle Background Worker Was the Most Expensive Thing I Owned"

Ashish Kr Jha · 2026-08-26T00:31:19.893Z

I opened the Cloud Run billing breakdown expecting the database to be the villain. Failing that, the AI generation, which calls out to a model and streams tokens back and feels expensive, in the way t

The min-instances=0 "fix" is the part worth flagging loudest, it's not a cheaper version of the same behaviour, it's silent data loss wearing a cost-optimization costume. Tasks pile up with zero errors anywhere, which means the failure surfaces as a support ticket about missing emails weeks later, not a billing alert.

The commit-before-enqueue ordering check is the detail I'd have missed entirely. Async execution accidentally papering over that bug via network latency, only to have eager mode "tear the paper off" immediately, is a sharp way to describe how a timing bug can hide behind a slow path for years and then surface the instant you make things faster.

The agent framing at the end is the most useful generalization though, "conventional" and "correct for this billing model" being different things, and the constraint living in a billing console the agent never had access to. That's not really a Celery story, it's a story about where the information an agent needs actually lives.

Good way to put it. The silent-failure part is what made that option especially dangerous, and the agent/context angle is the part I ended up taking away from it too.

This was such a relatable engineering story. It’s easy to overlook the cost of something that’s “just sitting there” until the numbers make you pay attention. I really liked the practical lesson here: infrastructure decisions should be revisited as usage and scale change. A great reminder that optimization isn’t always about making things faster it can also be about questioning what actually needs to be running in the first place.

Exactly. The useful question turned out to be less “what can I optimize?” and more “what actually needs to exist all the time?”

Eager mode is useful, but Cloud Run's request timeout is now the only bound your inline tasks have. A hung SMTP call that used to die at task_time_limit now sits on signup until the service timeout fires. Tune that number before a slow provider turns the hang into a 504 on signup.

Good catch. Once the task moves inline, the request timeout becomes part of the same failure budget. Bounded it at 5s per attempt and 12s total across attempts and backoff. The post also understates the normal case, since signup sends two emails sequentially, so that needs a correction too.

Search Hashnode

My Idle Background Worker Was the Most Expensive Thing I Owned

Responses(6)