The team set min_instances = 3 to eliminate cold starts on the AI workflow service. Reasonable thinking — LLM calls already add 5–15 seconds of latency, a 4-second cold start on the first request afte
Markdown supported · ⌘↵ to post
I’d also keep an eye on rate limits before pushing concurrency too high. It’s easy to make things faster on your side and just move the bottleneck to the model API.
min_instances=3 turning into 40% of the bill is the kind of fix that looks free on launch day and shows up a quarter later. Splitting cold start into provisioning, app init, and the first LLM call is the useful move, since the 2-5s of SDK plus pool warmup is the part you can actually attack. Did the warming endpoint alone get you there, or did you also lazy-load the heavier imports?
Cold starts are the hidden tax on serverless AI pipelines. Keeping a minimum instance alive changes the math completely.
Yunus Emre Vurgun
Writing about AI agents, hardware, and practical software.
One small addition to the warming endpoint: since /warm makes a real LLM call, it's worth excluding it from the request-count and latency dashboards (or tagging it), otherwise post-deploy warmups skew the p50/p95 you're using to tune min_instances. Also, Cloud Run's startup CPU boost pairs well with this: it shortens the 2–5s import/pool init for the cold starts that still happen after a scale-to-zero, without paying for an extra always-on instance.