---
title: "Discussion on \"Running LLMs on Kubernetes in Production: KServe, vLLM, llm-d, Envoy AI Gateway, and LiteLLM\" | Hashnode"
description: "Discussion on \"Running LLMs on Kubernetes in Production: KServe, vLLM, llm-d, Envoy AI Gateway, and LiteLLM\". One OpenAI-compatible endpoint, model-aware routing, GPU-aware scheduling, and scaling driven by inference pressure, not CPU"
url: "https://hashnode.com/posts/running-llms-on-kubernetes-in-production-kserve-vllm-llm-d-envoy-ai-gateway-and-litellm/6acb0e343177a7b8a6788104"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Running LLMs on Kubernetes in Production: KServe, vLLM, llm-d, Envoy AI Gateway, and LiteLLM](https://blog.shubhamtatvamasi.com/running-llms-on-kubernetes-in-production-kserve-vllm-llm-d-envoy-ai-gateway-and-litellm?utm_source=hashnode&utm_medium=feed)

One OpenAI-compatible endpoint, model-aware routing, GPU-aware scheduling, and scaling driven by inference pressure, not CPU

Oct 11 · 26 min read · blog.shubhamtatvamasi.com  
Updated Oct 25  
Read the article

-   [#kubernetes](https://hashnode.com/tag/kubernetes)
-   [#llm](https://hashnode.com/tag/llm)
-   [#vllm](https://hashnode.com/tag/vllm)
-   [#mlops](https://hashnode.com/tag/mlops)
-   [#ai](https://hashnode.com/tag/ai)

## Written by

[Shubham Tatvamasi](https://hashnode.com/@ShubhamTatvamasi)

@ShubhamTatvamasi

  
[View profile](https://hashnode.com/@ShubhamTatvamasi)

AI infrastructure engineer. Inference, GPUs and GitOps on Kubernetes.

## Responses

Markdown supported · ⌘↵ to post

No responses yet.
