---
title: "Discussion on \"Deploying AI Workflows on Cloud Run: Latency, Cold Starts, and Scaling\" | Hashnode"
description: "Discussion on \"Deploying AI Workflows on Cloud Run: Latency, Cold Starts, and Scaling\". Cold start mitigation, warming endpoints, concurrency for LLM calls, and latency profiling — deploying AI workflow services on Cloud Run correctly"
url: "https://hashnode.com/posts/deploying-ai-workflows-on-cloud-run-latency-cold-starts-and-scaling/6ac7e695eed20210cedb42bf"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Deploying AI Workflows on Cloud Run: Latency, Cold Starts, and Scaling](https://blog.madhav.dev/deploying-ai-workflows-on-cloud-run-latency-cold-starts-and-scaling?utm_source=hashnode&utm_medium=feed)

Cold start mitigation, warming endpoints, concurrency for LLM calls, and latency profiling — deploying AI workflow services on Cloud Run correctly

Oct 8 · 6 min read · blog.madhav.dev  
Read the article

-   [#cloud-run](https://hashnode.com/tag/cloud-run)
-   [#gcp](https://hashnode.com/tag/gcp)
-   [#fastapi](https://hashnode.com/tag/fastapi)
-   [#python](https://hashnode.com/tag/python)
-   [#ai](https://hashnode.com/tag/ai)
-   [#terraform](https://hashnode.com/tag/terraform)

## Written by

[Madhav Bhasin](https://hashnode.com/@madhavbhasin)

@madhavbhasin · Noida, India

  
[View profile](https://hashnode.com/@madhavbhasin)

Production patterns for backend engineers who are tired of hello-world tutorials.

Senior full-stack engineer and engineering lead with 5+ years building cloud-native systems at scale. I write about the things the docs don't cover — GCP, FastAPI, Terraform, event-driven architecture, and AI-powered backends. No hello-world demos.

## Responses 4

Markdown supported · ⌘↵ to post

[Yunus Emre Vurgun](https://hashnode.com/@yunusemrevurgun)

[Oct 15](https://hashnode.com/posts/deploying-ai-workflows-on-cloud-run-latency-cold-starts-and-scaling/6ac7e695eed20210cedb42bf/comment/6ad08fc31ce6f3599dfd3f31)

One small addition to the warming endpoint: since /warm makes a real LLM call, it's worth excluding it from the request-count and latency dashboards (or tagging it), otherwise post-deploy warmups skew the p50/p95 you're using to tune min\_instances. Also, Cloud Run's startup CPU boost pairs well with this: it shortens the 2–5s import/pool init for the cold starts that still happen after a scale-to-zero, without paying for an extra always-on instance.

[Julian Neagu](https://hashnode.com/@julianneagulab)

[Oct 12](https://hashnode.com/posts/deploying-ai-workflows-on-cloud-run-latency-cold-starts-and-scaling/6ac7e695eed20210cedb42bf/comment/6acc83d5823309e9e8dd45fe)

I’d also keep an eye on rate limits before pushing concurrency too high. It’s easy to make things faster on your side and just move the bottleneck to the model API.

[Kartik N V J K](https://hashnode.com/@kartik-nvj)

[Oct 9](https://hashnode.com/posts/deploying-ai-workflows-on-cloud-run-latency-cold-starts-and-scaling/6ac7e695eed20210cedb42bf/comment/6ac93d61e04056046d05ddf5)

min\_instances=3 turning into 40% of the bill is the kind of fix that looks free on launch day and shows up a quarter later. Splitting cold start into provisioning, app init, and the first LLM call is the useful move, since the 2-5s of SDK plus pool warmup is the part you can actually attack. Did the warming endpoint alone get you there, or did you also lazy-load the heavier imports?

[Puneet Khandelwal](https://hashnode.com/@lifestyle4u)

[Oct 9](https://hashnode.com/posts/deploying-ai-workflows-on-cloud-run-latency-cold-starts-and-scaling/6ac7e695eed20210cedb42bf/comment/6ac8c42a02eacf07b08571f8)

Cold starts are the hidden tax on serverless AI pipelines. Keeping a minimum instance alive changes the math completely.
