---
title: "Discussion on \"KAI Scheduler Deep Dive: How NVIDIA Schedules GPUs on Kubernetes\" | Hashnode"
description: "Discussion on \"KAI Scheduler Deep Dive: How NVIDIA Schedules GPUs on Kubernetes\". Queues, gang scheduling, fractional GPUs and topology-aware placement: how NVIDIA’s open-source scheduler shares GPUs"
url: "https://hashnode.com/posts/kai-scheduler-deep-dive-how-nvidia-schedules-gpus-on-kubernetes/6ad9850f035d0b8ec4de8b8f"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [KAI Scheduler Deep Dive: How NVIDIA Schedules GPUs on Kubernetes](https://blog.shubhamtatvamasi.com/kai-scheduler-deep-dive-how-nvidia-schedules-gpus-on-kubernetes?utm_source=hashnode&utm_medium=feed)

Queues, gang scheduling, fractional GPUs and topology-aware placement: how NVIDIA’s open-source scheduler shares GPUs

Oct 22 · 35 min read · blog.shubhamtatvamasi.com  
Updated Oct 25  
Read the article

-   [#kubernetes](https://hashnode.com/tag/kubernetes)
-   [#nvidia](https://hashnode.com/tag/nvidia)
-   [#gpu](https://hashnode.com/tag/gpu)
-   [#mlops](https://hashnode.com/tag/mlops)
-   [#machine-learning](https://hashnode.com/tag/machine-learning)

## Written by

[Shubham Tatvamasi](https://hashnode.com/@ShubhamTatvamasi)

@ShubhamTatvamasi

  
[View profile](https://hashnode.com/@ShubhamTatvamasi)

AI infrastructure engineer. Inference, GPUs and GitOps on Kubernetes.

## Responses 3

Markdown supported · ⌘↵ to post

[Abhinav Sharma](https://hashnode.com/@abhinavs1920)

[Oct 22](https://hashnode.com/posts/kai-scheduler-deep-dive-how-nvidia-schedules-gpus-on-kubernetes/6ad9850f035d0b8ec4de8b8f/comment/6ad9a6d0124f36abb02219ae)

The one-pod-at-a-time model is exactly why distributed training jobs stall on a busy cluster: part of the gang gets placed, the rest pend, and the placed pods hold GPUs idle. Treating the whole job as one scheduling unit fixes the utilization problem, not just the placement one.

[Abhinav Sharma](https://hashnode.com/@abhinavs1920)

[Oct 23](https://hashnode.com/posts/kai-scheduler-deep-dive-how-nvidia-schedules-gpus-on-kubernetes/6ad9850f035d0b8ec4de8b8f/comment/6adba150e3a04b494d2d64a9)

The Statement abstraction is the interesting design choice here: treating eviction scenarios as a transaction log with Checkpoint and Rollback keeps the real cluster untouched until a plan actually works. Pipeline being able to place a pod on resources still held by pods being evicted is what makes same-cycle reclaim practical. The stale-gang grace period is a nice touch too - evicting the surviving pods immediately after one OOM would thrash any busy cluster.

[arian rakibul](https://hashnode.com/@arianrakibul)

[Oct 22](https://hashnode.com/posts/kai-scheduler-deep-dive-how-nvidia-schedules-gpus-on-kubernetes/6ad9850f035d0b8ec4de8b8f/comment/6ad9f1488fc5a7111ccbcf67)

The separation between scheduling decisions and binding is particularly interesting. It improves resilience while making GPU allocation and scheduling failures easier to trace.
